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Chapter 1: Description of the Study 



Overview 

This report presents findings from the Massachusetts Family Child Care study, a two-year evaluation of 
the impacts of an early childhood education program on providers and children in family child care. The 
program — LearningGames 1 — is designed to train caregivers to stimulate children’s cognitive, language, 
and social-emotional development. The evaluation of LearningGames is one of four state experiments 
conducted as part of the Evaluation of Child Care Subsidy Strategies. The study is being conducted by 
Abt Associates Inc, with its research partners MDRC and the National Center for Children in Poverty of 
Columbia University, under a contract with the Administration for Children and Families within the 
U.S. Department of Health and Human Services. The goal of the evaluation is to provide information 
that states and local communities can use to inform their decisions about the use of child care subsidy 
and child care quality improvement funds. 

The objective of LearningGames is to promote children’s cognitive and language development through 
learning opportunities provided by their caregivers. LearningGames focuses on increasing the 
frequency of rich language interactions between caregivers and children. This emphasis grows out of 
the evidence of the importance of oral language development in children’s understanding of words and 
concepts, in their ability to become competent readers, and in their long-term academic success and of 
the role played by rich language stimulation in promoting children’s development. This evaluation of 
LearningGames examines the effectiveness of the program in changing the behavior of the family child 
care providers and the developmental outcomes for the children who are cared for by providers trained 
on LearningGames. The two major research questions for the study are: 

• Did LearningGames have significant positive impacts on the developmental support 
provided by providers to the children in care? 

• Did LearningGames homes have significant positive impacts on developmental outcomes 
for the children in care? 

The Massachusetts Family Child Care Study addresses important policy questions for Massachusetts 
and for other states about how to enhance the skills of the early education workforce to improve the 
quality of children’s experience in child care settings. Compared with center-based care, for which 
there is an expanding body of knowledge based on rigorous experimental research, family child care has 
been relatively neglected, except for descriptive studies and a small number of recent random- 
assignment studies, including Quality Interventions for Early Care and Education (QUINCE ) 2 and 



LearningGames is a series of early learning activities developed for the Abecedarian Project. MindNurture, a 
subsidiary of Teaching Strategies, Inc. created and currently disseminates the curriculum 
( http://mindnurture.com ). 

The QUINCE evaluation is a multi-state study of two assessment-based, individualized on-site consultation 
models: the Partnerships for Inclusion (PFI) consultation model, implemented in California, Iowa, Minnesota, 
Nebraska, and North Carolina, and the Rameys’ Immersion Training for Excellence (RITE) coaching model, 
implemented in Mississippi. The models consist of training for providers and teachers in both centers and 
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Project Great Start. 3 LearningGam.es was chosen to be implemented and tested in family child care 
because its approach seemed particularly well-suited to family child care providers. First, 
LeamingGames is organized around a set of 200 simple games that any adult (teacher, family child care 
provider, or parent) can learn to play with a child. Second, LeamingGames focuses on caregiver/child 
interactions in which there are no more than one or two children involved at a time. LeamingGames 
centers on intensive, one-on-one interactions as a platform that allows the adult to engage the child in 
meaningful conversation, to listen to the child and respond to the child’s questions and actions, and to 
scaffold and build on the child’s growing skills at using and understanding language. The one-on-one 
interactions in LeamingGames are also important for communicating to the child that he/she has an 
individual, caring and responsive relationship with the adult. Since home-based providers typically care 
for a small number of children, family child care appeared to be an environment in which caregivers 
could enact individualized, responsive relationships with each of the children in care. 

Another reason for choosing LeamingGames is that its precursor was the curriculum used in the 
landmark Abecedarian study. This study has shown a substantial and lasting impact of the intervention 
on children’s short- and long-term outcomes (Campbell, Ramey, Pungello, Sparling & Miller-Johnson, 
2002). While LeamingGames and its precursor have been studied in center-based care, home visiting 
programs and parenting programs, it has not been studied previously in family child care homes. 

The current study recruited family child care providers from all family child care networks in 
Massachusetts that met the study eligibility criteria. Eighteen of the 55 networks in the state were 
eligible. Within each network, individual providers were eligible for the study if they had been in 
operation for at least two years, were caring for at least one child under the age of 36 months, and were 
willing to comply with the requirements of random assignment. These criteria were intended to produce 
a sample of providers who were relatively stable, and therefore more likely to remain in operation over 
the two-year study period, and who cared for children who were young enough that they might be 
expected to remain with the provider for at least two years and could have extended exposure to 
LeamingGames. 

In each of the family child care networks, half of the providers who agreed to participate in the study 
were randomly assigned to LeamingGames and half were assigned to the control condition. All of the 
providers received the standard set of home visits from staff from the child care network which are 



homes, but has a special emphasis on providers in family child care homes, including license-exempt care. 

See http://www.fpg.unc.edu/~QUINCE/ . 

The Project Great Start Professional Development initiative is a research-based professional development 
program to increase the skills of center-based and home -based early childhood educators. The program offers 
educational experiences to providers through coursework and on-site coaching. A randomized study was 
designed to compare the impacts of coursework alone versus coursework with coaching versus neither 
(Neuman & Cunningham, 2009) A sample of 336 providers from four Michigan cities were randomly 
assigned to one of three conditions: coursework at a local community college in early literacy development; 
coursework plus 32 weeks of weekly coaching; and control. Coursework with coaching produced significant 
changes in provider practice (as measured by the Child/Home Early Language and Literacy Observation 
(CHELLO) and the Early Language and Literacy Classroom Observation (ELLCO)), while coursework alone 
did not. Coursework and coaching was very effective with home-based providers but not center-based 
providers. Finally, children had higher achievement in the settings in which the providers showed the largest 
gains in practice. 
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required of family child care networks funded by the state. 4 Network home visiting staff, selected by 
the family child care networks, were trained as LearningGcimes coaches, and these home visitors 
worked with the LearningGames providers for up to two years, offering training and mentoring during 
bimonthly home visits. Other home visiting staff worked with the providers in the control group, 
offering the usual ongoing training and technical assistance. The home visitors who were chosen to 
support the LearningGames providers were trained on LearningGames by the developer. This involved 
an initial three-day training, with quarterly one-day follow-up trainings and additional technical 
assistance and support. 

The study’s key research question is about the impact of LearningGames on children’s cognitive and 
language development. The study also estimates the impact of LearningGames on the behavior of 
family child care providers. The randomized design ensures a strong basis for answering these 
questions. Finally, the study addresses important questions about whether it is possible to train family 
child care providers to deliver such a program with fidelity and the level of support needed to 
accomplish this. 

The Massachusetts Family Child Care Study answers important policy questions for Massachusetts and 
for other states. Efforts to enhance the skills of the early education workforce are an important part of 
most states’ agendas for improving the quality of children’s experience in child care settings. Currently, 
states expend funds on initiatives to support and improve family child care, but have little guidance on 
what constitutes an effective intervention. The fact that family child care has been little represented in 
rigorous studies of programmatic interventions is of particular concern because of the large number of 
children who are cared for in home-based settings. Including all form s of home-based care, it is 
estimated that about half of all child care is provided in home-based settings (West, Wright, & Hausken, 
1996; Capizzano, Adams & Sonestein, 2000). The majority (60%) of children in the United States less 
than 6 years of age are in non-parental care; 35% of these children are being cared for in the homes of 
relatives and 22% in the homes of unrelated providers (National Center for Education Statistics, 2005; 
Halle et al., 2009). The percentage of children in home-based care settings is higher for infants and 
toddlers and for low-income and minority families than for preschool-age children and English- 
language speakers (Capizzano et al., 2000; NICHD ECCRN, 2004; Maher & Joesch, 2005). 

In addition, a recent study of early childhood education and care in Massachusetts indicates that the 
proportion of at-risk children (including children from low-income families and children from homes 
where Spanish is the primary language) is higher in family child care, compared with all forms of center 
care in the state (Rulf Fountain & Goodson, 2008). For example, the proportion of Massachusetts 
children from Hispanic backgrounds was 39% in family child care, compared with 23% in center care, 
and the proportion in family child care would most likely be higher if license-exempt family child care 
was included in the sample. The fact that home-based care represents such a large part of the universe of 
early childhood education, combined with the use of family child care among families whose children 
may be at increased risk for school readiness outcomes, makes it imperative that we attempt to develop 
strategies to ensure that these children are in care environments that promote learning and development. 



4 Networks in Massachusetts receive state hinds to provide technical assistance and training to their member 
providers, in part through regular visits by home visitors, as described in further detail later in this chapter. 
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The LearningGames Approach 



LearningGames grew out of an earlier parent-child curriculum that was developed and implemented as 
part of the Abecederian study. The LearningGames approach has five core components: 

• Approximately 200 games or activities for providers to use with one or two children at a 
time, covering the age range from birth to 5 years. The games are organized by the age of 
the child. Each game has written guidance for providers as well as a handout to guide 
parents on using these same games at home; the guides are available in Spanish as well as 
English. 

• Suggestions for providers on how to use “enriched caregiving” across all parts of the day, 
including during routine care and transition activities. 

• Suggestions for providers on how to engage in conversational reading with one or two 
children. 

• Training for providers on specific language priority strategies that build on children’s 
increasing cognitive and language sophistication. These include 3S (See, Show, Say) and 
3N (Notice, Nudge, Narrate). Conversation Books are provided as a platform for the 
provider to use to engage in these language priority strategies and in conversational reading 
with one or two children at a time. 

• Organizational plans and records to keep tracking of weekly planning and participation in 
LearningGames for each child in care. 

LearningGames. The 200 LearningGames are divided into five volumes by age (0-12 months, 12-24 
months, etc.). Each of the games is designed to support one or more specific development areas, 
including social emotional, early literacy, oral language, cognitive, and space and action. Each game is 
described on the front and back of a page. The front page provides the game’s name, a picture, and a 
brief overview. The reverse side provides more detailed information about what the adult and child 
should do and why the activity is important. Providers are encouraged to use the games with one or two 
children, to repeat them many times, and to deepen them as the child masters the game’s concepts. 

Enriched Careqivinq . The LearningGames approach also asks providers to incorporate activities 
throughout the day that enrich regular care routines. For example, for meal times, age-specific 
suggestions are provided such as: singing to a child during bottle feeding; talking about and naming 
items such as food, cup and spoon; using children’s names; pointing to and reading aloud letters on food 
labels; or writing names of needed items on grocery lists. It also provides other ideas such as naming 
things nearby, going for a walk, singing a song, and back-and-forth language play. 

Conversation Books . Read-aloud books are provided to each of the family child care providers. Each 
day, the intervention requires providers to read at least one book to each child; this book can be either a 
LearningGames read-aloud book or another children’s book. Providers are asked to read to one child 
alone or to two at a time, holding the child close to them while they read and encouraging children to 
respond, using the “3S Strategy” described below. 

Language Priority . The approach focuses on two sets of strategies, 3S (See, Show, Say) and 3N (Notice, 
Nudge, Nurture). “See, Show, Say,” draws out responses from children at three levels of difficulty and 
is tailored to the child’s age and abilities. Basing her choices on the child’s developmental stage, the 



Massachusetts Family Child Care Study: Evaluation of LearningGames 



4 



caregiver engages the child by identifying an object (“See”), having the child identify it through 
pointing (“Show”), and/or asking a question that will prompt the child to verbally identify the object 
(“Say”). 



The 3N strategy can also be woven into the day. The caregiver, observing the child, notices what the 
child is doing or preparing to do and uses words to describe it to the child or ask a question about it. 
Once the caregiver has noticed what a child is doing, she can then gently nudge the child toward a new 
learning opportunity (usually via a question). Finally, the provider narrates, telling the story of what 
the child is doing to increase the child’s awareness of the significance of his or her own actions. 

Planning and Record-Keeping Materials are provided so that family child care providers can make 
weekly plans for each child using the LearningGames approach and activities and track their 
educational progress in mastering each LearningGame. 

Implementation of LearningGames 

The implementation of LearningGames in family child care homes for the evaluation involved a 
complex set of organizational relationships among the state office responsible for the administration of 
child care subsidies, the child care resource and referral networks, the family child care networks and 
their providers, and the developer and trainers of LearningGames. Exhibit 1 . 1 summarizes the 
connections among these groups. The intervention initially was funded by the Massachusetts Office of 
Child Care Services (which was later integrated into a newly created Department of Early Education 
and Care). Two half-time study coordinators, who were staff of child care resource and referral agencies 
(CCR&Rs) provided support and technical assistance to network staff. Dr. Joseph Sparling, the 
program developer, and his staff provided materials and technical assistance to the project coordinators 
and the home visiting staff. Network home visiting staff, in turn, provided LearningGames support to 
providers selected for the intervention and ongoing standard technical assistance to those providers in 
the control group. 

Implementation of LearningGames in Massachusetts family child care homes relied primarily on a 
“train the trainer” approach. Selected home visiting staff from the participating family child care 
networks (networks are described in more detail in the section below) were trained by Dr. Joseph 
Sparling, the developer of LearningGames. The two half-time project coordinators from the CCR&Rs 
received funding to provide the network home visitors with ongoing support and technical assistance 
and attempted to maintain monthly contact with them. In the second year of the study. Dr. Sparling and 
his staff also offered additional direct training on LearningGames to participating family child care 
homes. 

In turn, home visiting staff were asked to provide training and support to providers on LearningGames 
during two visits each month. LearningGames providers are asked to do at least one LearningGame with 
each child under age 5 at least one time per each day but are encouraged to do them multiple times with 
each child each day. Working with home visitors trained in the approach, they are expected to identify 
the games that are most appropriate for an individual child, based on his or her age and developmental 
status, and decide when to move on to the next game in the series. They are to be encouraged by their 
home visitors to be creative, incorporating and deepening each game as time progresses. In addition, by 
enriching all caregiving and using the learning strategies, the LearningGames approach is to be woven 
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seamlessly throughout the child care day. A home in which LeamingGames is fully implemented is a 
language-rich environment in which a provider spends a substantial proportion of her time focusing on 
and interacting with one or two children at a time, and opportunities for children’s learning and 
development are provided throughout the day. 




Rationale for Testing LearningGames\w Family Child Care 

LeamingGames was selected to be implemented with family child care providers for several reasons. 
First, it seemed more appropriate for home-based providers compared with programs or curricula that 
were developed for center-based care. LeamingGames was adapted from a parent curriculum that was 
first used in the Abecedarian Program, a two-generational intervention in which parents and infants 
together attended an intensive education program at a center. Intervention children received full-time, 
high-quality educational intervention in a child care setting from infancy through age 5 in which each 
child received an individualized plan of educational activities designed as “games” that were 
incorporated into the child’s day. The activities, which were used by trained caregivers and the parents, 
focused on social, emotional, and cognitive areas of development and gave particular emphasis to 
language. The Abecedarian Program had substantial short-term impacts on infants (Ramey et. al., 2000; 
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Ramey & Campbell, 1984; Campbell & Ramey, 1994) and, in 20-year follow-up studies, has continued 
to show significant impacts on the children and their mothers (Campbell, Pungello, Miller- Johnson, 
Burchinal, & Ramey, 2001). LeamingGames is based on the curriculum used in the Abecedarian 
Program. Since the Abecedarian project, the curriculum has been adapted and used with center-based 
providers and with parents (reported in unpublished studies). 

The game-like content of LeamingGames was also expected to appeal to family child care providers, as 
well as the fact that there were materials for children from birth through preschool. This was crucial for 
providers who cared for mixed ages. Further, the LeamingGames focus on individualized work with 
each child was especially well-suited to the family child care environment, where the adult/child ratio is 
frequently much lower than in center-based settings. 

An important advantage of using LeamingGames as an intervention strategy in Massachusetts is that the 
training materials for providers and the guides for implementing the game-like activities for children are 
available in Spanish as well as English. The translated materials were especially important in light of 
the high proportion of Spanish-speaking providers in the sample and of children in care. 

The Massachusetts Context 

When recruitment for the study began in spring 2005, the position of OCCS commissioner was vacant. 
In July 2005, early childhood services in Massachusetts were reorganized and the state-funded child 
care programs became part of a newly created Department of Early Education and Care (EEC). A new 
commissioner of the EEC also assumed leadership on July 1, 2005. 

The study’s sample drew from licensed family child care homes affiliated with family child care 
networks. As is the case with all evaluations that do not rely on a representative sample, it is important 
to understand contextual information in order to assess the relevance of the study’s findings. Important 
factors to consider when assessing this study’s relevance for other family child care homes and for other 
states and communities include the program and policy environment and any unique features of the 
sample. In this case, family child care providers come from a state with fairly high regulations and, 
within that state, drew from a group that receives relatively high levels of support and technical 
assistance. 

Below we provide a brief overview of Massachusetts family child care licensing and regulation, the 
functions of Massachusetts family child care networks, and information about the quality of care. In 
Chapter 2, we provide information about what is known about the sample. 

Massachusetts Family Child Care Licensing Policies 

Massachusetts has a relatively higher degree of regulation of family child care homes than is the case in 
many other states. The Commonwealth requires family child care to be licensed if one or more unrelated 
children are in care, and the maximum number of children allowed in a home -based setting is 10. Most 
states set the minimum threshold at three or four unrelated children and only one other state has the 
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maximum threshold as low as does Massachusetts. 5 In order to be licensed, family child care providers 
must, among other requirements, be at least 1 8 years of age and have three credits in child development 
and at least 9 months of experience in a child care program. Providers must meet additional 
requirements related to education and/or years of experience if they care for more than seven children. 
Providers are required to use a curriculum that is developmentally and linguistically appropriate, 
supports school readiness and includes goals for knowledge and skills to be acquired by children in a 
range of topics including language development, math, and science. 

Massachusetts Family Child Care Networks 

The study recruited family child care homes from among the 55 family child care networks funded by 
the Massachusetts Office of Child Care Services. There are approximately 8,600 licensed family child 
care homes in the state and approximately 32% of them (about 2,700) belong to these networks. Most 
of the children whose family child care is supported by subsidies are placed with family child care 
homes that are affiliated with the Commonwealth’s family child care systems. Approximately 40% of 
all licensed family child care homes care for children who receive child care subsidies; of these, nearly 
80% belong to systems and the remaining 20% are unaffiliated. 

One major benefit of system affiliation for family child care homes relates to child care subsidies. 
Networks coordinate the process of placing children who are eligible for subsidies in the affiliated 
homes. They also coordinate all of the child care reimbursements so that the family child care home 
providers do not interact directly with the EEC for payment. 

In addition to facilitating the subsidy process for affiliated homes, family child care networks must offer 
their affiliated providers a range of technical assistance. They are required to have a home visitor 
assigned to each home who must make at least monthly visits. Home visitors must have at least a high 
school degree, 9 credits of early childhood education, and between 9 and 36 months of child care 
experience, depending on educational attainment, or they must have a Child Development Associate 
(CD A) certificate. The home visitors are required to assess and evaluate the needs of enrolled children 
and their families, help providers develop individualized and appropriate curricula, provide resources 
and support for work with children and families, identify providers’ professional development needs and 
associated professional development plans, and generally provide technical assistance. 

In turn, family child care providers have a written provider agreement with the network that makes clear 
the expectations in terms of program hours and days of operation, recruitment of full fee-paying private 
children, professional development requirements, attendance and training and reimbursement assistance 
for attending training, and the terms under which the network will manage provider billing and fee 
collection. 

Networks must assess the professional development needs of affiliated providers on an annual basis. 
They must ensure that the family child care providers are offered mentoring, career counseling, 
academic advising, course, classes and conferences that offer hours that can go toward annual in-service 
training requirements or approved continuing education units (CEUs). For all of these efforts, networks 



5 For all information about family child care regulations, please see the Child Care Licensing Study of 2007 
(National Child Care Information Center & National Association for Regulatory Administration, 2009.) 
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receive approximately $10/day per each child in the affiliated providers for whom child care is 
subsidized. 



At the time of the study, there were 55 family child care networks, located throughout the state. The 
catchment areas of the networks are not contiguous; more than one network can and often does serve the 
same local area, especially if it is relatively urban, and may actively recruit the same family child care 
homes. 

A priori, it was decided that the sample of networks for the study would be drawn from those systems 
that met two conditions necessary to participate in the study. These included that they must have at 
least two full-time home visitors so that the visitor trained to do LearningGames would not visit homes 
in the control group, believing that it was highly likely that the LearningGames training would influence 
home visitors’ technical assistance to all providers, resulting in promoting LearningGames practices 
with providers in the control group. In addition, systems needed to have in place, as a standard practice, 
the expectation that each provider would receive two home visits per month. These eligibility criteria 
were intended to ensure that systems had the capacity to implement the intervention with providers at 
the planned level and would also rule out the possibility that an effect of the LearningGames 
intervention was the result of additional home visits instituted for LearningGames providers only. 
Eighteen of the 55 networks met these criteria, representing approximately one third of affiliated family 
child care homes in the state. 

In the winter prior to the beginning of the study, the Massachusetts Office of Child Care Services 
surveyed the networks to determine, among other things, the number of home visiting staff, the number 
of homes assigned to each visitor, and the standard number of home visits. Exhibit 1.2 provides 
information on characteristics of the networks that responded to the survey as well as the networks that 
met participation conditions. Forty-five (80%) of the 55 networks returned the survey, including 16 of 
the 1 8 networks that met the selection criteria. The networks that met the study selection criteria were 
larger, on average, with 47 affiliated homes versus 38 for the broader state sample. A larger proportion 
of the networks that were eligible to participate served more than 50 homes (38% versus 26% of the 
statewide sample) and, not surprising, the eligible networks employed more home visitors on average 
(2.5 for the eligible agencies; 1.9 for all agencies). 

Slightly more than half of all agencies (51%) said it was their standard practice to do two visits per 
month to each provider in their network. Since this was an eligibility criterion for the study, all of the 
eligible networks reported that they conducted two home visits per month. The number of homes 
assigned to home visitors in the entire group and the subset of eligible agencies was similar (19 versus 
20 homes per visitor). However, 44% of the subset of agencies reported caseloads that were less than 
15 homes per home visitor, compared with 56% of the entire group of networks. (It should also be noted 
that caseload information is difficult to inteipret in that home visitors may have other duties as well.) 

A number of network staff, including home visitors and family service specialists, food and nutrition 
coordinators, and others, were reported to make regular visits to the providers. Networks reported a 
broad range of educational attainment among their home visiting staff. They reported that 1 6% of 
named home visiting staff had a high school degree and 31% had a bachelor’s degree (Exhibit 1.3). 
Educational attainment of home visiting staff from the networks eligible for the study is similar to that 
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among providers from all networks combined, with the exception that a higher percentage of staff in the 
eligible networks had college degree (31% for all versus 34% for qualified networks). 



Exhibit 1.2: Characteristics of Massachusetts Family Child Care Networks 3 




All Family Child Care 
Networks in 
Massachusetts 
(n=43) 


Family Child Care 
Networks Eligible for 
Study 
(n=16) 


Average number of affiliated homes 


38 


47 


Percent serving less than 25 homes 


42% 


25% 


Percent serving more than 50 homes 


26% 


38% 


Average number of home visitors 


1.9 


2.5 


Percent in which home visitors conduct 2 
visits per month 


51% 


1 00% 


Average number of homes assigned to each 
visitor 


20 


19 


Percent of networks in which home visiting 
caseload is < 15 


56% 


44% 


Source: Survey of family child care networks conducted by the Massachusetts Office of Child Care Services, 2005. 
a 12 of the 55 networks in the state did not respond to the survey; of these, 2 of the 18 eligible networks did not respond 
to the state survey. 



Exhibit 1.3: Highest Level of Educational Attainment of Home Visiting Staff a 




Home Visiting Staff From 
All Family Child Care 
Networks in Massachusetts 
(n=1 12) b 


Home Visiting Staff From 
Networks Qualified for the 
Study 
(n=65) c 


High school 


16% 


12% 


High school and some college 


20% 


22% 


Associates degree and/or CDA 


25% 


28% 


Bachelor’s degree 


31% 


34% 


Advanced degree 


8% 


3% 


Source: Survey of family child care networks conducted by the Massachusetts Office of Child Care Services, 2005. 

a 12 of the 55 networks in the state did not respond to the survey; 2 of the 18 eligible networks did not respond to the 
state survey. 

b 7 missing responses. 

c 5 missing responses. 
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The Quality of Family Child Care in Massachusetts 

There have been several studies that have assessed family child care quality in Massachusetts, although 
none occurred during the period when the evaluation took place. One of them used the Family Day Care 
Rating Scale (FDCRS), which was also used to evaluate a subset of homes that participated in 
Massachusetts Family Child Care Study. 

The most recent is the Study of Universal Pre-Kindergarten in Massachusetts (Rulf Fountain & 

Goodson, 2008). The study assessed the quality of a sample of programs for which the majority of 
children were paid for by subsidies, using a sample selected proportionately to their representation 
among programs in the state serving a majority of subsidized children. The study assessed quality using 
the Classroom Assessment Scoring System (CLASS) (Pianta, La Paro, & Flamre, 2008). The CLASS 
produces ratings across three broad domains of provider/child interaction considered critical for learning 
and development: Emotional Support, Classroom Organization, and Instructional Support. The study 
found that family child care homes in Massachusetts that serve at least 50% subsidized children are 
rated highly on the quality of Emotional Support for children, where their average rating is 5.6 out of 7. 
They also are rated highly on Classroom Organization (4.9 out of 7). Flowever, the homes received 
much lower ratings on Instructional Support (1.8 out of 7). 

The second major study is the Massachusetts Cost and Quality Study, conducted in 2001-2002 
(Marshall et al., 2003). For that effort, data were collected for a representative sample of Massachusetts 
family child care homes. The study relied on the Family Day Care Rating Scale (FDCRS) and the 
Global Caregiving Rating Scale to measure environmental quality and the sensitivity and quality of the 
provider’s relationship with children. Two subscales most relevant to the current evaluation are the 
Language and Reasoning Scale, which is a measure of the use of language in the setting and the 
opportunities for learning about language that are provided for children; and the Learning Activities 
Scale, which measures the types and variety of activities available for children and how the daily 
activities are scheduled and supervised. Among the family child care providers in the Massachusetts 
Cost Quality Study, the average subscore for Language and Reasoning was 4.57 and the average score 
for Learning Activities was 4.41, in both cases falling between the rating categories of “adequate” and 
“good.” The study found that the average FDCRS rating was 4.39, also falling within that category. 
Other studies have used the FDCRS to assess the quality of family child care homes in other states and 
communities with global FDCRS scores that fall between 4 and 5 (see, for instance, Bromer, Van 
Haitsma, Daley, & Modigliani, 2009; Raikes, Raikes & Wilcox, 2005.) 

Conceptual Model and Research Questions 

The evaluation centers on the impact of LearningGames on children’s developmental outcomes. The 
study begins with the hypothesis that LearningGames will affect children through changes in the 
behavior of their family child care provider. Changes in provider behavior are hypothesized to occur 
under specific conditions: if the level of training and support offered to the family child care providers is 
sufficient, their behavior and interactions with children will change; ultimately, these changes will result 
in positive impacts on children’s socio-emotional, cognitive and language development. This pathway 
from curriculum to child outcomes will depend on providers reaching a sufficiently intensive level of 
implementation of the curriculum (sometimes called fidelity of implementation) to effect changes in 
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children, as well as on children being in care a sufficient length of time to receive the benefit of their 
provider’s enhanced instructional and caregiving strategies. 

Exhibit 1.4 (page 16) lays out a conceptual model for the effects of LeamingGames. The model shows 
that, at heart, LeamingGames is a provider training model. That is, the LeamingGames program of 
structured activities and behavior guidelines (2 nd column in model) is hypothesized to lead to improved 
child outcomes by changing and enhancing the ways that providers interact with the children in their 
care (3 rd column in model). As this column indicates, the LeamingGames objectives for provider 
behavior focus on the quality of provider/child verbal interactions. Providers are trained to engage in 
more language interactions overall as well as language interactions that build children’s language skills 
and their grasp of new concepts and new vocabulary. Providers are trained that the most effective 
language interactions with children are extended, involve rich language structures and vocabulary, and 
are with individual or pairs of children rather than larger groups. The overall quality of the 
provider/child relationship is also a focus of LeamingGames. Having a responsive, caring adult is 
assumed to be a factor in children’s learning and development and learning. Finally, LeamingGames 
focuses on the importance for providers to weave the rich, responsive interactions with children 
throughout the day, during routines and transitions as well as during structured learning activities. 

The left-hand column in the model depicts the professional development model that was designed and 
implemented in the Massachusetts study. LeamingGames itself does not specify a particular design for 
professional development. The model in Massachusetts was designed to be an effective approach given 
(a) the level of resources available for professional development, (b) the organization of the family child 
care system in the state, and ( c) the geographic spread of the sample of family child care homes. That 
is, the “train-the-trainer” approach allowed the study to make use of existing staff at the family child 
care networks, which, in turn, saved resources and enabled the study to provide in-person coaching to 
homes that were spread across the state. 

The 4 th column in the model shows the child outcomes that are expected to be affected by 
LeamingGames. For children from birth through 5 years of age, LeamingGames specifies age-related 
cognitive developmental goals, such as concept development (from the earliest concepts such as object 
permanence to expanded concepts such as numbers, colors, and time) and acquisition of vocabulary 
(from simple words and phrases up to sophisticated language structures). For children from 3 to 5 years 
of age, the goals include additional early literacy skills and use of language to express their own ideas 
and needs. Further, across the age span, there are age-appropriate goals in the area of socio-emotional 
development, starting with secure attachment as a base for exploration and learning, and leading to the 
developmental of understanding of emotions in self and others, emotion control, and social skills with 
adults and peers. 

The study’s three major research questions flow from this model: 

• Were the elements of the provider training model implemented as planned? 

— Did providers receive the level of training and support that was assumed to be 
required for high-quality implementation of LeamingGames by the family child 
care providers? 

— Was there evidence of high-fidelity implementation of LeamingGames by the 
providers ? 
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• Did the LeamingGames training result in meaningful differences between LearningGames 
providers and control providers in their interactions with the children in care? 

• Did children in the LeamingGames homes have better developmental outcomes, compared 
with children in control homes? 

For the questions about impacts on providers and children, the study focused on a subset of the 
outcomes shown in the full logic model. For providers, the study focused on three major outcomes: 
amount of rich language interactions between the provider and the children in care, level of provider 
support for children’s comprehension of concepts and vocabulary, and level of responsiveness of the 
provider to the children in care. For child outcomes, the study looked at language development for 
children across the age range. For the 3- to 5-year-olds, the study also assessed acquisition of basic 
concepts that are part of school readiness. The study did not assess children’s socio-emotional 
development. The rationale for why particular outcomes were studied and why other outcomes were not 
includes limitations in available measures and statistical concerns about multiple comparisons. These 
are discussed in more detail in Chapters 2 and 4. 

The remaining chapters of the report are as follows: 

• Chapter 2 presents the study methodology, including the timeline, recruitment process, 
random assignment, baseline sample size, sample attrition, and data collection measures 
and schedule. 

• Chapter 3 describes the implementation of LeamingGames in the family child care homes 
in Massachusetts, including the planned professional development model, how it was 
actually implemented, and barriers to full implementation of the planned model. It then 
provides information about provider fidelity in implementing the approach. 

• Chapter 4 presents the findings on the impacts of LeamingGames on family child care 
providers. 

• Chapter 5 provides a discussion of the findings and conclusions. 
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Exhibit 1.4: Logic Model for Impacts of LearningGames on Providers and Children 



Professional 
Development Model 



• Training and TA by 
LearningGames developer 
for project coordinators and 
for home visitors 

• Training and TA by home 
visitors for LearningGames 
homes 

• Direct training (Year 2) by 
developer for 
LearningGames homes 



— 



LearningGames 
Program Model 



• Weekly planning for 
LearningGames activities 
for each child in care 

• LearningGames activity 
every day with every child 

• Interactive book reading 
every day with every child 

• Consistent use of enriched 
caregiving 

• Use of 3S and 3N 
strategies every day with 
every child in context of 
LearningGames activities, 
interactive reading, and 
enriched caregiving 

• Parent LearningGames 
materials and 
conversations books sent 
home weekly 



Provider Outcomes 



• Individualized oral language 
interactions with each child 

• Language interactions with 
children that involve content 
rich in terms of concepts, 
vocabulary 

• Verbal and non-verbal 
interactions with children that 
are extended and meaningful 

• Consistent support across 
activity contexts for children’s 
comprehension of new 
concepts, vocabulary, early 
literacy skills 

• Development of warm, 
responsive relationships with 
each child 

• Enriched caregiving 
throughout day, even during 
routines, transitions 

• Frequent interactions with 
individual/pairs of children 



Child Outcomes 



Children < 3 years 

• Understanding basic concepts 
(spatial concepts, simple 
descriptive concepts, body 
parts) 

• Understanding of specific 
words, phrases 

• Sense of security and 
attachment through 
relationship with caring adult 



Children 3-5 years 

Receptive and expressive 
vocabulary knowledge 
Early literacy skills— print 
knowledge, print motivation, 
phonological awareness, 
alphabet 

Understanding of concepts 
such as color, shapes, 
quantity, numbers, time 
Use and understanding of 
expanded linguistic structures 
Socio-emotional development 
(understanding of emotions, 
emotional control, social skills) 



Context 

Overall level of quality of homes at baseline 










Chapter 2: Study Methodology 



Overview 

The study followed the implementation and outcomes of LeamingGam.es over approximately two years. 
This chapter describes the overall design of the study, including the study timeline; the processes for 
recruitment and randomization of providers; the sampling design and sample attrition; and the 
methodology for assessing the implementation process and the outcomes for both family child care 
providers and the children in their care. 

Study Design 

The study was a randomized cluster design, with family child care providers assigned to treatment 
( LearningGames ) or control (business as usual). Children were clustered within provider, so that all 
children in treatment homes had an opportunity to receive LearningGames and all children in control 
homes received their provider’s regular program. The study is intended to be an effectiveness study, in 
that the impact of LearningGames was studied under typical or real-world conditions. That is, 
LearningGames was implemented in the family child care homes the same way it would have been in 
the absence of the study: staff from the family child care networks provided most of the training on 
LearningGames and providers were expected to integrate LearningGames into their regular program. 
Further, although the study tried its best to achieve high fidelity of implementation across all treatment 
providers through training and ongoing support, variation in implementation was expected. Since the 
study was being conducted under real-world conditions, providers had control over whether and how 
well they implemented the components of the intervention. 

Study Timeline 

Exhibit 2.1 summarizes the schedule of major activities in the LearningGames study, which will be 
described in more detail in following sections. Implementation was funded by the state in spring 2005 
and the costs for materials and initial training had to be incurred during the state fiscal year that ended 
that June; therefore the schedule of recruitment, random assignment, and initial training was accelerated. 

Those providers who agreed to be part of the study were randomly assigned to either LearningGames or 
a control group. Initial training of home visiting staff (described in more detail in Chapter 3) occurred 
in midsummer 2005. Also during summer 2005, baseline data on the participating providers were 
collected. Starting in fall/winter 2005, professional development support was given to LearningGames 
providers, which lasted for two years (also described in Chapter 3). In fall 2006, Abt conducted one- 
year provider observations and determined, with ACF’s approval, that LearningGames was not being 
implemented strongly enough to merit child assessments at that time. In December 2007, the two-year 
provider observations were conducted, and children in the homes were assessed over an 8-week period, 
December 2007-January 2008. 
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Exhibit 2.1: Study Timeline 





Spring 2005 


Summer 2005 


Fall 2005 


Winter 2005 


Spring 2006 


Summer 2006 


Fall 2006 


Winter 2006 


Spring 2007 


Summer 2007 


Fall 2007 


Winter 2007 


Recruitment of family child 
care networks and homes 






Random assignment of 
providers 




Baseline data collection 








LearningGames intervention 
and support 








First year provider 
observations 




Second year provider 
observations 








Child assessments 







Sampling Plan 

Sample Recruitment 

Recruitment of providers was a two-stage process: first, family child care networks were recruited and 
second, providers within the participating networks were recruited. Each is described in more detail 
below. 

Family Child Care Networks 

As described in Chapter 1, 18 of the 55 family child care networks with contracts with the state met the 
initial selection criteria and were recruited for the study. Recruitment of networks was a joint 
responsibility of the state and the study team. The Massachusetts Office of Child Care Services sent a 
letter to these family child care networks, asking them to participate in the study and inviting them to a 
meeting at which Abt staff could explain the evaluation and answer any questions. Abt then followed up 
with telephone calls and meetings with interested networks. Of the 18 networks invited to participate, 16 
initially agreed to do so. 

Networks that joined the study agreed to a number of conditions, including that they would: 

• Help recruit their providers for the study and commit to attempting to get at least 1 0 family 
child care homes to participate; 

• Allow the providers who consented to participate to be randomly assigned to either receive 
LearningGames or to continue to receive standard technical assistance; 

• Contribute the time of home visiting staff to be trained in LearningGames', 

• Agree that home visitors would train family child care homes using the Learning Games 
professional development protocols, including conducting two home visits per month; 
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• Ensure that home visitors trained in LeamingGames would not visit providers in the control 
group (which meant that systems would likely need to change home visitor assignments); 
and 

• Help negotiate data collection requirements with providers (e.g., family child care 
observations and child assessments). 

In return, all of the LeamingGames materials and training were provided at no change to networks or 
providers. 

The 16 networks signed and returned partnership agreements that outlined the above conditions and 
specified the roles and responsibilities of Abt and of the network. Prior to random assignment of 
providers, one of the agencies that had agreed to participate withdrew because it was going through a 
reorganization, reducing the number from 16 to 15. 

A total of approximately 1300 homes were affiliated with the participating agencies, representing almost 
half of all affiliated family child care homes and a little more than one-third of the state’s family child 
care providers who care for children supported by subsidies. Exhibit 2.2 compares 13 of these 15 
participating networks that responded to the Massachusetts Office of Child Care Services survey with 
16 of the 18 qualifying networks that responded to the survey. On average, the participating networks 
served slightly a larger number of providers (51 versus 47 homes), and fewer networks served 25 or 
fewer homes (19% versus 25% of the sample). While the average reported home visitor caseloads were 
similar; 3 1% of the participating networks reported caseloads that were lower than 15 homes per visitor; 
compared to 44% for all qualifying networks. 



Exhibit 2.2: Characteristics of Family Child Care Networks in the Eligible Networks and in 
the Study 




Family Child Care 
Networks Eligible for 
Study 
(n=16) a 


Participating Family 
Child Care Networks 
(n=13) b 


Average number of affiliated homes 


47 


51 


Percent serving fewer than 25 homes 


25% 


19% 


Percent serving more than 50 homes 


38% 


38% 


Average number of home visitors 


2.5 


2.5 


Percent that do 2 home visits per month 


100% 


100% 


Average number of homes assigned to 
each visitor 


19 


20 


Percent for which home visiting caseload 
is fewer than 15 homes 


44% 


31% 


a 18 networks were eligible for the study, but two networks did not respond to the state survey. 
b 15 networks agreed to participate in the study, but two of these did not respond to the state survey. 
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The largest network in the study included seven regional offices in different parts of the state, each with 
separate administrative and home visiting staff. For the purposes of the study, the separate regional 
offices were treated as networks for the purposes of implementation and random assignment. The result 
is that the sample included 22 networks and/or regional offices. 

Exhibit 2.3 provides information about the educational attainment of the home visiting staff of networks 
eligible for the study and participating networks and indicates that the staff in both groups had similar 
levels of educational attainment. 



Exhibit 2.3: Highest Level of Educational Attainment of Home Visiting Staff in the Eligible 
Networks and in the Study a 




Home Visiting Staff From 
Family Child Care 
Networks Eligible for the 
Study 
(n=65) a 


Home Visiting Staff From 
Family Child Care 
Networks Participating in 
the Study 
(n=57) a 


High school 


12% 


16% 


High school and some college 


22% 


22% 


Associates degree and/or CDA 


28% 


31% 


Bachelor’s degree 


34% 


27% 


Advanced degree 


3% 


3% 


a 5 missing responses for staff listed by networks that responded to the survey. 



The networks selected the home visitors to be trained in LearningGames from their current home 
visiting staff. Networks were not required to select these home visitors at random. LearningGames 
home visiting staff were similar to other home visitors in terms of educational attainment and years of 
experience. Flowever, it is possible that any provider impacts found by the evaluation are due to 
differences between the LearningGames and other home visitors in areas such as interpersonal skills, 
knowledge of child development, motivation or other factors that could affect the quality of the home 
visits. 

Family Child Care Homes 

After networks agreed to participate, family child care homes in each network were recruited for the 
study, a process that occurred over a two-month time period. To be eligible, a provider had to have 
been in operation for at least two years and had to have at least two children in care who were less than 
36 months of age. 

Recruiting these homes was a joint activity between the networks and the Abt team. The recruitment 
approaches were as follows. Typically, Abt attended a network’s scheduled meeting of family child 
care providers, where staff described the study, answered questions, and distributed English and Spanish 
brochures providing details about the study and what participation entailed. At these meetings, Abt staff 
distributed a sign-up sheet for providers who were interested in participating. In some cases, Abt staff 
attended more than one meeting at a particular network. If there was only one meeting with a group of 
providers, Abt staff then followed up with network staff, who checked again with interested providers 
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and collected signed provider consent forms. Again, these forms provided details about the study and 
what would be required of those participating in the evaluation. If there were two meetings, Abt 
collected consent forms at the second meeting. Many of the networks had a substantial number of 
Spanish-speaking family child care providers. Whenever possible, Abt bilingual staff attended the 
meetings and did translation but sometimes network staff did the translation. In some cases, a network 
preferred that its own staff recruit providers for the study. In these cases, Abt met with network staff 
and provided them with information and materials about the study and answered any questions. In turn, 
network staff recruited the providers and collected the provider consent forms. 

At the outset, some networks decided how many homes could feasibly participate in the study. Some 
networks that served multiple communities limited recruitment to specific neighborhoods or 
communities because it was logistically possible to have at least two home visitors provide services in 
that area (one for the LeamingGames providers and one for providers in the control group). Others had 
to exclude Spanish-speaking providers from the intervention because only one of their home visitors 
spoke Spanish and therefore by necessity would have had to visit both the treatment and control 
providers. Appendix A shows the final sample of 353 homes by their network or regional office 
affiliation. 

One of the questions of interest before the intervention began was whether the family child care homes 
were operating at a level of quality that could support the kinds of individualized, high-quality verbal 
interactions that are the cornerstone of LeamingGames. Although a quality criterion was not used as 
part of the eligibility criteria, data were collected at the end of the study to describe the quality of the 
providers that were part of the impact analyses at the end of two years of intervention. Quality of the 
settings was measured using the Family Child Care Environmental Rating System — Revised (FCCERS- 
R) (Harms, Cryer & Clifford, 2007), a well-known measure of the quality of family child care settings. 
The measure provides an assessment of the overall quality of the family child care setting. The 
FCCERS-R includes 38 items that form seven subscales: Space, Listening and Talking, Activities, 
Interactions, Program Structure, Personal Care Routines and Parents and Provider. Each subscale 
receives a rating from 1 to 7. Scores of 2 and lower are considered below minimal quality; scores 
between 3 and 4 are considered minimal quality; and scores between 6 and 7 are considered good to 
excellent quality. 

The FCCERS-R was administered in a randomly selected subset of 60 of the family child care homes 
(30 LeamingGames homes and 30 control group homes) when data were collected in winter 2007. The 
five FCCERS-R subscales that focused on provider-child interactions were completed (Space, Listening 
and Talking, Activities, Interactions and Program Structure). Two subscales, Personal Care Routines 
and Parents and Provider, were not administered. The average scores for the control group homes on 
four of the five subscales were in the minimal range (Exhibit 2.4). Only the score for “Interactions” was 
in the good-to-excellent range. At the same time, more than 40% of homes were rated in the good-to- 
excellent range on Listening and Talking and Program Structure. Based on the results of the FCCERS-R 
ratings of the LeamingGames control homes, we concluded that the quality of the homes was high 
enough to afford a platform for implementing the LeamingGames approach. 
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Exhibit 2.4: Rating of Quality of LeamingGames Homes on Selected Subscales of the Family 
Child Care Environment Rating System — Revised Edition (FCCERS-R) 


FCCERS-R Subscales 


% Homes 
Below Minimal 
(<2) 


% Homes 
Minimal 
(3-4) 


% Homes 
Good- 
Excellent 
(5-7) 


Average 
Rating 
(out of 7) 


Space 


15.5% 


55.2% 


29.3% 


4.1 (1.2) 


Listening and Talking 


15.5 


43.1 


41.4 


4.3 (1.7) 


Activities 


41.4 


53.4 


5.2 


3.3 (1.0) 


Interactions 


15.5 


17.2 


67.2 


5.2 (1.9) 


Program Structure 


25.9 


29.3 


44.8 


4.2 (1.8) 


All Subscales 


15.5 


67.2 


17.2 


4.2 (1.2) 


Sample includes 25 bilingual and 25 English-speaking control providers from the control group sample. 



To help evaluate the meaning of the quality ratings of homes in the LeamingGames study, these ratings 
were compared to quality ratings of a sample of Massachusetts family child care homes from the 
Massachusetts Study of Child Care Cost and Quality (Marshall et al., 2003) described briefly in Chapter 
1. In the 2003 study, the Family Day Care Rating Scale (Harms & Clifford, 1989), the precursor to the 
FCCERS-R, was administered in a sample of 203 homes randomly selected from all licensed family 
child care in the state. 6 Four of the subscales on the FCCERS-R had comparable subscales on the earlier 
version of the measure. Quality ratings on two of these four subscales — Space and Listening and 
Talking — were similar across the two samples, in terms of the percentage of homes with good-to- 
excellent quality (Exhibit 2.5). The percentage of homes with good-to-excellent quality for Activities 
was lower among the homes in the study sample, while the percentage for Interactions was higher for 
homes in the study sample. 

Random Assignment 

The random assignment of family child care homes was conducted within the 22 family child care 
networks, meaning that approximately half the homes affiliated with each were assigned either to the 
LeamingGames group or the control condition. This within-network random assignment ensured that 
the samples of LeamingGames and control providers were equivalent in terms of network 
characteristics that might be related to the study outcomes and also did not result in overly burdening 
some networks, which may have been assigned more LeamingGames homes than they could feasibly 
support. 



6 



The sample of homes was drawn from across Massachusetts, proportional to the region’s share of the state- 
licensed homes. 
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Exhibit 2.5: Quality Ratings 3 for Two Samples of Family Child Care Homes in Massachusetts 


Subscales 


% Homes 

Below Minimal (< 2) 


% Homes 
Minimal (3-4) 


% Homes 

Good-Excellent (5-7) 


Average Score 


Learning 

Games 

FCCERS-R b 


MA 

Cost/Quality 

FDCRS c 


Learning 

Games 

FCCERS-R b 


MA 

Cost/Quality 

FDCRS c 


Learning 

Games 

FCCERS-R b 


MA 

Cost/Quality 

FDCRS c 


Learning 

Games 

FCCERS-R b 


MA 

Cost/Quality 

FDCRS c 


Space 


15% 


23% 


55% 


46% 


29% 


31% 


4.1 


4.1 


Listening and Talking 


15 


13 


43 


47 


41 


40 


4.3 


4.6 


Activities 


41 


14 


54 


50 


5 


36 


3.3 


4.4 


Interactions 


15 


14 


17 


39 


67 


47 


5.2 


4.6 


“Quality ratings on four comparable subscales from two versions of the family child care quality rating scale (FCCERS-R and FDDCRS). 
b Sample includes 60 bilingual and 25 English-speaking providers from the Massachusetts LearningGames study. 
c Sample includes 203 randomly selected family child care homes in Massachusetts. 




Random assignment was conducted by Abt Associates in spring 2005. The agencies provided Abt 
with the names of participating providers, and providers were assigned to treatment or control using a 
computerized assignment program. The process resulted in 173 homes being assigned to 
LeamingGam.es and 180 assigned to the business-as-usual control group. Letters that notified each 
family child care home were prepared and sent to the family child care agencies. Agency staff then 
distributed the letters personally to the providers. At the same time of random assignment, 
approximately 1250 children were enrolled in the 353 study homes. 

Sample Attrition 

Provider Sample 

The sample suffered from substantial attrition between baseline and the posttest, two years after the 
intervention began. Attrition in the provider sample derived from two sources: individual providers 
who dropped out of the study (provider-level attrition) and entire agencies that dropped out (agency- 
level attrition). Since random assignment was conducted within agencies, agency attrition resulted in 
the loss of approximately equal numbers of treatment and control providers. Consequently, loss of an 
agency does not bias the sample, but it does reduce the power of the analyses to detect impacts. 
Provider-level attrition, on the other hand, both reduces power and potentially introduces bias into the 
sample. 

The implementation of the LearningGames training started more slowly than anticipated, as described 
in further detail in Chapter 3. Implementation is considered to have started in fall 2005, 4-6 months 
after random assignment. After one year of implementation (fall 2006) when the first round of family 
child care observations occurred, the overall attrition rate was 43% (Exhibit 2.6). The rate of attrition 
was similar for treatment and control providers. About half of the attrition was the result of two 
agencies dropping out of the study (82 providers). These agencies dropped from the study because the 
level of burden to provide the LearningGames intervention was more than anticipated and because 
home visitors were meeting with resistance among their homes assigned to the treatment group to 
using the LearningGames approach. Several other networks in the study served family child care 
homes in the same communities that were served by the two agencies that dropped out. (Chapter 4 
provides additional information about the characteristics of the provider sample at baseline and the 
samples remaining at each data collection point.) In addition, during that year, 68 individual 
providers dropped out of the study. While Abt tried to collect information about reasons for dropping 
out, it was not consistently reported. In general, reasons given by individual providers for dropping 
out included leaving the family child care network or moving, no longer caring for children, and not 
wanting to participate in the fall 2006 observation. 

At the end of the two years of implementation of the LearningGames provider training (fall/winter 
2007), the overall attrition rate was 58% and two more agencies had dropped out of the study. There 
was higher attrition among the LearningGames providers, compared with the control providers (60% 
versus 55%). About half of the attrition from the provider sample was the result of the four agencies 
dropping out (108 providers). The remaining attrition was the result of 95 providers who withdrew 
for individual reasons. The provider-level attrition varied widely across the 18 agencies remaining in 
the sample after two years. Four of the agencies lost at least half of their providers, while other 
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agencies lost less than 10% of their sample. (See Appendix A for a description of level of attrition by 
agency.) 



Exhibit 2.6: Provider Sample and Attrition Over the Study 




Fall 2006 

(18 months post RA) 


Fall/Winter 2007 
(30 months post RA) 


T 

(n = 173) 


C 

(n = 180) 


Total 
(n = 353) 


T 

(n = 173) 


C 

(n = 180) 


Total 
(n = 353) 


Providers remaining 


98 


105 


203 


69 


81 


150 


Overall attrition 


43.3% 


41.7% 


42.5% 


60.1% 


55.0% 


57.5% 


Sources of attrition 














Agency attrition 3 


21.3% 


25.0% 


23.2% 


28.9% 


32.2% 


30.6% 


Individual provider attrition 


22.0% 


16.7% 


19.3% 


31.2% 


22.8% 


26.9% 


a Two agencies dropped out of the study by fall 2006; two additional agencies dropped out by fall 2007. 



The level of provider attrition from the sample was substantial, even after one year of the 
intervention. In addition, by the end of the study, there also was differential attrition for the 
LeamingGames providers and the control providers. Overall attrition is an issue for power to detect 
impacts, and it could affect the generalizability of the findings. If the providers who are left in the 
sample represent a particular subset of the original sample of providers, then the results can only be 
generalized to this subgroup. The differential attrition poses a potential threat to the internal validity 
of the estimates of the impacts on providers. 7 The level of attrition argues for a set of analytic steps to 
be conducted as part of the impact analyses. First, the study needs to determine if there are 



7 Under the guidelines for the What Works Clearinghouse (WWC), the rating given to a randomized study 
(“Meeting Evidence Standards,” “Meets Standard with Reservations,” or “Fails to Meet Evidence 
Standards”) is based on overall attrition differences in the rates of attrition for the intervention and 
comparison groups. Both overall and differential attrition contribute to the potential bias of the estimated 
effect. The WWC has developed a model of attrition bias to calculate the potential bias; a combination of 
overall and differential attrition rates are considered to generate acceptable, potentially acceptable, and 
unacceptable levels of expected bias that are defined for and applied consistently for studies in a topic area. 
In randomized cluster studies, attrition is examined at both the cluster and the individual subject levels. For 
the LeamingGames study, the differential attrition is not a problem. The attrition levels for both providers 
and children by the end of the study are considered to be severe, which means that the study would be 
judged as “expected to result in an unacceptable level of bias even under optimistic assumptions, and the 
study can receive a rating no higher than Meets Evidence Standards with Reservations, provided that it 
establishes baseline equivalence of the analysis sample” (p. 14). That is, the WWC requires that RCTs with 
high levels of attrition present evidence that the intervention and comparison groups are alike. 
Demonstrating equivalence minimizes potential bias from attrition that can alter effect size estimates. 
Baseline equivalence of the analytical sample must be demonstrated on observed characteristics, using 
these criteria: The reported difference of the characteristics must be less than 0.25 of a standard deviation 
(based on the variation of that characteristic in the pooled sample) and the effects must be statistically 
adjusted for baseline differences in the characteristics if the difference is greater than 0.05 of a standard 
deviation (What Works Clearinghouse, 2008). 
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significant baseline differences between the analytic samples of treatment and control providers. If 
there are differences, this calls for controlling for baseline characteristics in the impact analyses. 

Since the study collected baseline date on provider characteristics and behavior, these data can be 
used for these adjustments. 

Child Sample 

Child outcomes were assessed in winter 2006. To be eligible for the child assessments, a provider 
had to have at least one child enrolled who met three criteria: age (at least 12 months of age and not 
yet in kindergarten), time in care (at least 6 months in care with provider), and parent permission for 
at least one child to participate in the assessments. In addition, the provider herself had to agree to be 
part of the study. As shown in Exhibit 2.6, the level of provider attrition in fall/winter 2007 was 58%, 
which translates into 150 providers. Of these 150 providers, only 121 met all of the eligibility criteria 
for the child assessments. This means that the sample of providers with children in the assessments 
represented 34% of the original sample of providers (Exhibit 2.7). This level of cluster attrition 
introduces a potentially high level of bias into the child impact analyses. As was true for the analysis 
of provider impacts, the attrition requires the study to examine baseline differences between the 
remaining treatment and control children and to adjust for any differences by using baseline 
covariates in the impact analyses. 



Exhibit 2.7: Provider Attrition at Time of Child Assessments 






Winter 2008 

(34 months post RA; 2 years of implementation) 




T 


C 


Total 




(n = 173) 


(n = 180) 


(n = 353) 


Providers remaining 


59 


62 


121 


% of sample 


34.1% 


34.4% 


34.3% 


Children assessed 


182 


192 


374 


% < 36 months 


57.6% 


57.0% 


57.3% 



The child assessments took place in January 2008, which was 30 months after random assignment 
and just over two years after full implementation of the intervention. At that time, 150 providers 
remained in the sample. Across these homes, only 4.5% of the children who were present at baseline 
were still in care two years later (Exhibit 2.8). As would be expected, the highest proportion of 
children remaining in the homes was the group who, at baseline, were less than 2 years of age. 
Among the older children, most were no longer in care two years after baseline, since these children 
had reached the age of school entry. The story was similar for the providers who participated in the 
child assessments. Among these 122 homes, only 5.4% of the children who were assessed had been 
in the homes for two years, and only since the time of random assignment. 
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Exhibit 2.8: Proportion of Children in Care at Baseline and at Two Years 



Age at Baseline 


Proportion Children Remaining in 
Care at 24 Months 
(n = 150 providers) 


Proportion Children Remaining in 
Care at 30 Months 
(n = 122 providers) 


0-12 months 


13.3% 


6.7% 


13-24 months 


7.9 


16.3 


25 - 36 months 


7.3 


12.5 


27 - 48 months 


1.2 


2.7 


49 - 60 months 


1.4 


1.9 


School-age 


1.0 


1.1 



Even for the small number of the children who were assessed who had been in care at baseline, the 
study does not have child-level baseline data to examine differences between treatment and control 
children at baseline or to adjust for any differences that do exist. 8 Therefore, the study team 
concluded that the child assessment sample cannot be assumed to support credible analyses of the 
impact of LeamingGcimes on children. Before reaching that conclusion, impact analyses were 
conducted on the children who were assessed. In the interest of transparency, we have presented the 
results of these analyses in Appendix B. 

Data Collection 

Data obtained for the study came from multiple sources. To address questions on implementation, 
information came from interviews with providers, home visitors, network administrators, and 
LearningGames trainers; review of tracking documents of technical assistance activities; and ratings 
of the fidelity of implementation of LearningGames conducted by home visitors as well as by study 
staff. To address the research question on provider impacts, baseline provider outcomes were 
measured through direct observation by the agency home visitors and outcomes after Y ear 1 and Y ear 
2 by independent study staff. To address the research question on child impacts, children were 
assessed individually on standardized tests, which were administered by independent study staff. The 
data collection plan is described below for the evaluation of implementation and of impacts on 
providers, including the measures, training, and data collection procedures. Appendix B provides the 
same information for the evaluation of impacts on children. 

Implementation Outcomes 

Two components of implementation were evaluated: the implementation of the training/support 
model for the LearningGames trainers and the providers, and the implementation of the 
LearningGames program model in the family child care homes. Each is discussed below. 



Home visitors were asked to assess children in the homes at baseline using Ages and Stages. By the end of 
summer 2005, a small percentage of assessments had been done. It was clear that requiring home visitors to 
complete children’s baseline assessments would further delay the implementation of the intervention so this 
requirement was dropped. 
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Implementation of the Professional Development Model 

The implementation of the planned training and support model (i.e., the “professional development” 
model) for LeamingGames was documented through different sources: interviews with staff and 
other stakeholders involved in the implementation; and review of tracking documents maintained by 
the project coordinators, who were hired to provide ongoing technical assistance and support to home 
visiting staff and their networks. Tracking documents about the implementation of LeamingGames 
were collected for the two years. Documents included monthly monitoring reports from project 
coordinators and fidelity tracking sheets submitted to the LeamingGames developer by home visitors. 

In addition, in summer 2007, Abt staff also conducted interviews with family child care network staff 
(including directors and home visitors) from networks participating in the study, the LeamingGames 
developer, study coordinators, and lead staff at the Massachusetts Department of Early Education and 
Care. In total, 38 individuals associated with implementation of the LeamingGames study were 
interviewed. These included all staff who had a leadership role in the study (the developer and his 
technical assistance staff, study coordinators, and lead staff at the Massachusetts Department of Early 
Education and Care), as well as staff from selected networks that had experienced both relatively high 
and relatively low levels of provider attrition. Interviews included questions aimed at better 
understanding the intended implementation of LeamingGames and how the curriculum was 
ultimately used. In addition, respondents were specifically asked questions about their individual 
backgrounds, about the organizations for which they worked and about their roles in the study (e.g., 
their general involvement in the LeamingGames study, changes in their responsibilities resulting 
from study participation, and the nature and quality of training and technical assistance they 
received). They were also asked about the planning for the study and their perceptions of different 
aspects of the study’s implementation. 

Evaluation of the Implementation of LeamingGames in the Homes 

Implementation of the LeamingGames program model was measured from two perspectives. First, 
we gauged the extent to which treatment providers adhered to the practices deemed key to 
LeamingGames by the developer. This was accomplished through ratings of provider practices by 
the LeamingGames home visitors, using a 1 0-item checklist created by the developer (the 
“Procedural Fidelity Form.”) Second, we used the observations that were conducted on both 
treatment and control providers to assess provider practices, regardless of the details of the 
LeamingGames program. We created a Fidelity Scale based on selected items from among the 
measures used in the provider observations (as described below) that represented practices that 
aligned with LeamingGames and compared practices across treatment and control providers. 9 The 
scale is based on 1 1 of these items, which were recoded on a 3-point scale, where 1 = behavior not 
exhibited or exhibited infrequently by provider, 2 = behavior exhibited occasionally by provider, and 
3 = behavior exhibited often or consistently by provider. Each provider was rated on all 1 1 items. 

The total score on the Fidelity Scale could range from 1 to 33 points. 10 



9 Please see Exhibit 3.4 in Chapter 3 for the actual scale. 

10 Three of the items were conditional on age of child. If there were no children in the home in the age range 
referenced in the item, a provider was not scored. The final score on the fidelity measure was calculated as 
a mean across the number of valid items for each provider (ranging from 9 to 1 1 items). 
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Provider Outcomes 



Provider outcomes were assessed through direct observations of the providers and children in both the 
LearningGames and control homes. The measures are described following the discussion of the data 
collection protocol. 

Data Collection Protocol 

Observations of the homes were conducted at three times over the intervention period: at baseline 
(spring 2005), after one year of implementation of LearningGames (fall 2006), and after two years of 
implementation (fall/winter 2007). The impact analyses focused on the winter 2007 observations, 
which represented the end of the intervention. At this time point, all treatment group providers 
remaining in the sample had been in the study for more than two years, and treatment providers had 
received at least 24 months of systematic training support on LearningGames from the network home 
visitors. (The range of exposure varies somewhat, depending on the month providers received their 
initial training.) 

Different observation measures were used at the three time points, and different groups were 
responsible for conducting the observations (Exhibit 2.9). 

• At baseline, staff from the family child care networks conducted observations to 
document the quality of the home environments prior to implementation of 
LearningGames. The measures used in these observations were the QUEST (Goodson, 
Layzer & Layzer, 2005) and the Arnett Caregiver Interaction Scale (Arnett, 1987). 

• After one year of implementation, independent trained study staff conducted observations 
of treatment and control homes, to evaluate both preliminary impacts on provider 
behavior and to assess whether the level of implementation of LearningGames had 
reached a sufficiently strong level to justify conducting child assessments. Four 
observation measures were used. Two were adapted from the Observation Measures of 
Language and Literacy Instruction (OMLIT, Goodson et al., 2005) — the Snapshot of 
Activities, and the Read Aloud Profile. The Read Aloud Profile was used during a 
structured read aloud that providers conducted for the purposes of the study, using study- 
provided picture books. A third measure was an existing instrument, the Arnett 
Caregiver Interaction Scale (Arnett, 1989). The observers also re-administered the 
QUEST. 

• After two years of implementation, independent trained study staff again conducted 
observations of treatment and control homes, to evaluate the end-of-intervention impacts 
of LearningGames on providers. Four observation measures were used, and three of the 
four were the same as were used at the one-year observations. The two OMLIT measures 
were used, although the Snapshot was adapted for this observation and the Read Aloud 
Profile was used only if a read aloud occurred naturally during the observation. The 
Arnett Caregiver Interaction Scale was administered again. The fourth measure 
administered was the TALK (Goodson & Layzer, 2008), which was newly developed for 
the study to assess the amount and quality of provider oral language with individual 
children. 
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• As part of the observations after two years of implementation, the study team 

administered the Family Child Care Rating System-Revised (Harms et al., 2007), to 
asses the overall quality of the treatment and control homes using a standard measure of 
quality. The measure was used in a subset of the homes. 

Measures 

The observation measures are described below. 

QUEST . The QUEST consists of two parts: an Environment Checklist, which rates the resources and 
safety of the care setting, and the Caregiver Rating Scale, which assesses the behavior of the adult 
who is caring for the children in six areas: caring and responding, supporting social-emotional 
development, supporting play, supporting cognitive development, supporting language development 
and early literacy, and television and computers. For this study, only the Caregiver Rating Scale was 
used, based on the appropriateness of the items to the objectives of the LearningGames approach. The 
QUEST Caregiver Rating Scale has 69 items; the observer rates the provider on each item using a 4- 
point Likert scale. The QUEST was used at baseline and at the end of the first year. Several items on 
the QUEST, most closely aligned with the LearningGames implementation, were also collected in fall 
2007. 



Exhibit 2.9: Observation Measures of Providers and Family Child Care Homes at Three 
Time Points 


Observation Measure 


Observation Time Point (Observers) 


Spring 2005: 
Baseline 

(child care agency 
staff) 


Fall 2006: One Year 
of Implementation 
(study staff) 


Fall/Winter 2007: 
Two Years of 
Implementation 
(study staff)) 


QUEST Caregiver Rating Scale 


X 


X 


9 items only 


Arnett Caregiver Interaction 
Scale 


X 


X 


X 


OMLIT Snapshot of Activities 




X 

(adaptation #1) 


X 

(adaptation #2) 


OMLIT Read Aloud Profile 




X 

(structured 

situation) 


X 

(natural 

occurrence) 


TALK 






X 



OMLIT . The Observation Measures of Language and Literacy Instruction (OMLIT) (Goodson et al, 
2006) is a battery of measures developed originally to assess language and literacy instruction in 
group settings. The OMLIT measures aspects of early childhood education practice which, based on 
professional opinion and research, support children’s acquisition of early literacy skills. The OMLIT 
also provides general descriptive information about the organization of and activities in the care 
setting. While the full OMLIT battery includes six measures, two were selected and adapted for 
family child care environments: 
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• The Snapshot of Activities (OMLIT-SNAP) is a time-sampled description of child 
activities and groupings, integration of literacy in other activities, and language in the 
setting. It has two sections. The Environment section describes the number of children 
and adults present, as well as the type of adult (staff, parent). The Activities section 
describes activities that are taking place. Then, for each activity, the observer records the 
number of children and adults in that activity, whether any adult or child is talking, 
whether they are speaking English or another language, whether any literacy materials 
are used (text, writing, letters), and if there is singing with the children (distinguished on 
the measure because of its potential as a phonological awareness/oral language support). 

• The Read Aloud Profile (OMLIT-RAP) is a description of adult behavior when reading 
aloud to children. The RAP records adult behavior during the read-aloud session on 
supports for comprehension, questions, attention to print knowledge, and vocabulary. 

The RAP also includes quality indicators which summarize particular aspects of the read- 
aloud: (1) the degree to which the adult introduces and contextualizes new vocabulary to 
support children’s learning, (2) the extent to which the adult uses open-ended questions 
that invite children to engage in prediction, imagination, and/or rich description, and (3) 
the quality of any post-reading book-related activities that the adult organizes (beyond 
oral discussion). 

For this study, both of the OMLIT measures were adapted to be appropriate for family child care and 
to emphasize the variables that align most directly with the objectives of the LeamingGames program 
(copies of the adapted coding forms for the Snapshot and the RAP are included in Appendix C). The 
Snapshot records the activities and groupings of all children and adults present in the family child 
care at the time of the observation. Each child and adult is assigned to one of 14 activities, such as 
reading, math, computer, snack, and the like. If the provider is involved in an activity with one or 
more children, her language and overall level of engagement are coded. Provider involvement is 
categorized on a scale ranging from” “observing”,” to “managing” up to “playing 
with/teaching/demonstrating/discussing” with children. For each activity, the Snapshot indicates not 
only the number of children in that activity but also their ages (infants, toddlers, or preschoolers). 

The data can then be analyzed in terms of proportion of time over an observation period that children 
spent in each type of activity, actively engaged with the provider, playing alone or with other children 
and their ages. During the observations, the Snapshot was completed every five minutes. 

The OMLIT RAP did not require substantial adaptation for family child care, since it was designed to 
describe the behavior of a single adult reader with any number of children. The RAP records the 
provider’s interactions with children during a read-aloud session outside of the actual reading of the 
text in the book. This includes comprehension supports (e.g., telling the story in advance), open- 
ended questions, attention to conventions of print, focus on print knowledge, and introduction of new 
story-related vocabulary and types of definitional supports for new vocabulary (pictures or props, 
definitions with synonyms or antonyms, semantic networks). Codes were added to the RAP to record 
specific features of LeamingGames such as the read-aloud strategies. A RAP was completed each 
time that the provider read aloud to children during an observation period. 

In previous studies, the reliability of the OMLIT measures, as assessed by inter-rater agreement, was 
above .80 for the major codes. Reliability of the observers on the LeamingGames study was assessed 
as part of the training procedure and is described below. 



Massachusetts Family Child Care Study: Evaluation of LeamingGames 



29 





TALK . The TALK was developed for the LearningGames study to assess the extent to which 
providers engage in extended conversation with individual children, as well as other types of one-on- 
one language interactions that could build children’s oral language skills. The TALK codes 
provider/child verbal interaction in five categories: management or helping, provider only (including 
provider narrating child’s actions), simultaneous verbalization (singing, chanting, rhymes), discussion 
(short, fewer than 4 turns back and forth), and extended discussion (4 or more back and forth turns, 
with provider building on child’s responses). In administering the TALK, the observer rotates 
through the children present, observing each one for two 5-minute segments and recording any 
individual language interactions with the provider during those two 5-minute periods. Six TALK 
observations were completed in an hour of observation (five minutes of observation and coding, 
following by a Snapshot as described above). This schedule meant that, in most homes, children were 
observed with the TALK at least three times. 

Arnett Caregiver Interaction Scale (CIS) . The 26-item Caregiver Interaction Scale assesses the 
quality and content of the caregiver’s interactions with children. The scale was designed to provide 
information on various socialization practices that have been identified in research on parenting. The 
scale can be used without modification in both center and home-based settings. The items measure 
the emotional tone, discipline style, and responsiveness of the caregiver in the setting. The observer 
rates the extent to which the caregiver exhibits the behavior described in the item on a 4-point scale, 
ranging from not at all (1) to very much (4). Averages can be calculated for each subscale. The items 
are usually organized into the following four subscales: (1) positive interaction (warm, enthusiastic, 
and developmental^ appropriate behavior), (2) punitiveness (hostility, harshness, and use of threat), 
(3) detachment (uninvolvement and disinterest), and (4) permissiveness. The CIS has adequate 
psychometric properties. In terms of internal consistency, Layzer, Goodson & Moss (1993) obtained 
Cronbach alphas of .91 for warmth/responsiveness (positive interaction) and .90 for harshness 
(punitiveness), while Resnick and Zill (1999) obtained alphas for the total scale of .98 for lead 
teachers in early childhood classrooms and .93 for assistant teachers. Jaeger and Funk (2001) reported 
coefficients of .81 and higher for the sensitivity (positive interaction), punitiveness, and detachment 
subscales. In addition, Jaeger and Funk reported inter-rater reliability coefficients ranging from .75 to 
.97 between a certified observer and trainees. 

At posttest, nine items from the QUEST were adapted and added to the CIS. The items focus on 
provider behavior that aligns more closely to the objectives of LearningGames, including extended 
interactions (verbal or nonverbal) with individual children, enrichment of daily routines with 
language or learning, language-rich interactions with children, encouragement of reading, 
encouragement of exploration. 

Observer Training 

Prior to the observations of the family child care homes, observers were trained to reliability on the 
observation measures by senior Abt staff familiar with the measures and with protocols for training 
observers. Data collectors were trained on the three observation measures over a 4-day period. 
Training on all the observation instruments combined an item-by-item discussion of the instruments, 
illustrations and discussion of items using video-recorded segments of adult-child interactions and 
book reading, and procedures for conducting observations in family child care homes. Trainees were 
also given opportunities to practice independently coding video-recorded segments of adult-child 
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interactions and book readings. Among the 12 trainees, 10 passed reliability testing for all the 
observation instruments. To demonstrate reliability, trainees (a) achieved 80% agreement with expert 
raters from Abt Associates on written vignettes of family child care environments; (b) achieved 80% 
agreement with expert raters from Abt Associates on independently coded video-recorded samples of 
interactions and book readings, and (c) achieved 80% agreement with a trainer when both the trainer 
and trainee conducted observations in family child care homes that were not part of the study sample. 
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Chapter 3: Implementation Study 



Overview 

The Massachusetts Family Child Care Study is the first time that LeamingGames has been 
implemented on a wide scale in family child care. Therefore, the study offered the opportunity to 
learn about the feasibility of implementing the program in home-based child care settings with high 
fidelity to the model, as well as learn about factors that were either “enablers” or “disablers” of 
successful implementation. The study examined two major components of the implementation of 
LeamingGames : (a) the implementation by the developers and home visitors of the professional 
development model for training and supporting providers on LeamingGames, a model that was 
designed specifically for the Massachusetts study, and (2) the implementation by providers of the 
LeamingGames program with the children in their care. The major research questions that guided the 
implementation study are: 

• What was the professional development model as planned, i.e., what types and amounts 
of training and support were providers in tended to receive? 

• To what extent was the professional development model implemented as planned, and, 
for areas where the planned model was not fully implemented, what were the factors that 
hindered full implementation? 

• What was the LeamingGames program model as planned, i.e., what types and amounts 
of activities were providers intended to use with their children? 

• To what extent was the LeamingGames program model implemented as planned, and, 
for areas where the planned model was not fully implemented, what were the factors that 
hindered full implementation? 

In this chapter, we first describe the program model and the professional development model as 
planned. The professional development model includes specification of the planned roles and 
responsibilities of the organizations involved in the implementation of LeamingGames and the types 
and amount of support to be provided to the family child care homes and providers. The program 
model includes the components of the LeamingGames approach. We then provide information about 
the extent to which the implementation of the study reflected fidelity to the major elements of the 
professional development and program models. 

The implementation study indicates that the professional development model was only partially 
implemented. Because of a series of barriers to implementation that were encountered, the quality of 
support to family child caregivers provided by home visiting staff was not optimal, at least in the 
initial period of the intervention. In addition, it appears that home visitors did not consistently 
provide the specified amount of LeamingGames technical assistance time. 

Despite the inconsistency in the implementation of the planned professional development for 
providers, there was evidence that providers were using the LeamingGames program model. The 
home visitors reported that many of the providers used and incoiporated the LeamingGames 
strategies in their daily practices. In addition, Abt’s fidelity measure, drawing from its provider 
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observations, indicated that there was a modest difference between LearningGames and control 
providers in the developmental practices supported by LearningGames. More details are provided in 
the following sections. 

The Massachusetts LearningGames Program Model 

The LearningGames approach includes several components: 

• LearningGames activities for providers’ use with individual or pairs of children; 

• “Enriched caregiving” across all parts of the day, including during routine care and 
ordinary activities; 

• Interactive reading using LearningGames Conversation Books and other storybooks; 

• Specific language priority strategies to support children’s language and cognitive 
development: 3S (See, Show, Say) and 3N (Notice, Nudge, Narrate); 

• Parent handouts to encourage use of the same LearningGames activities at home that 
providers are using in the care setting; and 

• Supporting materials, including a manual for family child care providers, and 
documentation and organizational plans and records. 

Each is described in further detail below. 

LearningGames 

The 200 LearningGames are divided into five volumes corresponding to each year of age from birth 
through age 5 (e.g., 0-12 months, 13-24 months). Each game is designed to support one or more 
specific development areas, including social emotional, early literacy, oral language, cognitive, and 
space and action. As an example, Exhibit 3.1 provides information on the games in the third volume 
(for months 25-36) and the developmental areas that they address. The set of games in each volume 
are expected to take approximately one year to complete, if the child and caregiver starts with the first 
game in the volume. (Providers and children choose the most developmentally appropriate game with 
which to begin and this game may be midway through a volume.) The games increase in their 
developmental sophistication so that the last game in a volume is designed to be appropriate for a 
child is who is approximately one year older than when he or she played the first game. As shown in 
Exhibit 3.1, in the third volume of the curriculum, 24 of the games address at least one socio- 
emotional domain, 1 1 games address early literacy, 14 address oral language development, 19 address 
at least one cognitive development domain, and 6 address space and action. 
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Exhibit 3.1 : Developmental Themes Addressed by LeamingGames Volume 2 (Games 68- 
100+) Designed for Children Between 2 and 3 Years of Age 




Social and Emotional 


Early Literacy 


Oral Language 


Cognitive 


Space & Action 


Game 


Needs & Feelings 


Self-Image 


Independence 


Sharing & Cooperation 


Family & Culture 


Knowing & Classifying 


Creativity 


Object Permanence 


Visual Motor 


68. Showing On Part 












• 


• 


• 




• 






69. Making Faces 


• 


• 








• 






• 








70. Family Circle Games. 


• 






• 


• 














• 


71 . Dress Up 




• 


• 












• 








72. Playing With a Mirror 


• 


• 














• 




• 




73. Seeing It a New Way 






• 










• 


• 




• 




74. What’s Your Name? 




• 






• 


• 














75. Chanting Nursery Rhymes 








• 


• 




• 












76. Showing “One” and “Two” 














• 


• 






• 




77. Color Sorting 














• 


• 






• 




78. Building Blocks 






• 












• 




• 




79. Showing Your Needs 


• 






• 






• 












80. Making Ox and Xs 












• 




• 






• 


• 


81 . Playing With Others 








• 










• 






• 


82. Choosing the Doll’s Clothes 




• 










• 












83. Drawing Around Things 






• 






• 










• 




84. Whispering 








• 






• 












85. Happy Face, Sad Face 


• 


• 






• 






• 










86. Two Together 








• 










• 






• 


87. Choosing and Stringing 








• 














• 




88. In, Out, and Around 














• 


• 








• 


89. Giving One to Each 








• 


• 


• 








• 






90. Making a Fun Path 






• 








• 










• 


91 . Using Words for Time 












• 


• 


• 










92. Listening and Supporting 


• 






• 


• 
















93. Pairing and Sorting Pictures 












• 


• 


• 






• 




94. What’s Gone 














• 


• 




• 
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Exhibit 3.1 : Developmental Themes Addressed by LeamingGames Volume 2 (Games 68- 
100+) Designed for Children Between 2 and 3 Years of Age 




Social and Emotional 


Early Literacy 


Oral Language 


Cognitive 


Space & Action 


Game 


Needs & Feelings 


Self-Image 


Independence 


Sharing & Cooperation 


Family & Culture 


Knowing & Classifying 


Creativity 


Object Permanence 


Visual Motor 


95. Cutting and Pasting 






• 












• 




• 




96. Helping Him Help Himself 






• 
















• 




97. What Would Happen If... 












• 


• 


• 


• 








98. Running & Walking Together 








• 






• 












99. Telling Family Stories 




• 




• 


• 


• 




• 










100. 1 See Something That Is 














• 


• 










100+ One Picture, Two Labels 








• 


• 


• 


• 


• 










Total 


6 


7 


7 


12 


8 


11 


16 


14 


9 


3 


11 


6 


Source: Sparling & Lewis (2001). 



Each of the rows in Exhibit 3.1 represents a title of a LeamingGame in the third volume of the 
approach. Each game is described on the front and back of a perforated page that can be removed 
from the LeamingGames volume and used separately or displayed. The front side provides the 
game’s name, a picture, and a brief overview. On the bottom of the front page, there is a short answer 
to “why” this is important. For instance, for “Showing One and Two” (Game 76) the game says that 
the puipose of the game is “to show that ‘one’ and ‘two’ tell a particular amount.” The picture shows 
and adult and child together doing the activity. The front page overview gives a three or four 
sentence explanation of the game. The reverse side provides more detailed information about what the 
adult and child should do and why the activity is important in terms of enhancing children’s 
development and provides tips for extending and deepening the game. 

Providers are trained to select a game that would be developmentally appropriate for each child in 
care. The provider repeatedly uses each game with one or two children, deepening the game as 
children master the game’s concepts. For instance, LeamingGame #50 (for 12-24 months of age) is 
called “First Nesting.” The caregiver gives the child two objects that can be nested, such as two 
different size cups and lets the child explore how they fit together, while narrating what the child is 
doing and, eventually helping if necessary. In subsequent “plays” of the game, the caregiver might 
find other objects around the house and let the toddler explore how they can be “nested” while 
narrating what the toddler is doing. After a game has been mastered, the provider selects the next 
game, with the expectation that there will be natural overlap between games. 
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For each of the games, providers are asked to distribute handouts to parents that guide parents in 
using the same games at home. The parent handouts are in English and in Spanish. 

Enriched Caregiving 

The LecirningGames approach also asks providers to incoiporate activities throughout the day that 
enrich regular care routines, including meal preparation and eating, putting on coats to go outside, 
tying shoes, and other routine care. For example, for meal times, age-specific suggestions are 
provided such as: singing to a child during bottle feeding; talking about and naming items such as 
food, cup and spoon; using children’s names; pointing to and reading aloud letters on food labels; and 
writing names of needed items on grocery lists. It also provides other ideas such as naming things 
nearby, going for a walk, singing a song, and back-and-forth language play. 

Conversation Books 

As part of the materials, read-aloud books are provided to each of the family child care providers. 
These mostly are brightly colored board books composed of pictures and simple concepts. Each day, 
the intervention requires providers to read at least one book to each child; this book can be either a 
LecirningGames read-aloud book or another children’s book. Providers are asked to read to one child 
alone or to two at a time, holding the child close to them while they read and encouraging children to 
respond, using the “3S” strategy described below. 

Language Priority Strategies 

The approach focuses on two sets of strategies: 3S (See, Show, Say) and 3N (Notice, Nudge, 

Narrate). “See, Show, Say” draws out responses from children at three levels of difficulty and is 
tailored to the child’s age and abilities. Basing her choices on the child’s developmental stage, the 
caregiver engages the child by identifying an object (“See”), having the child identify it through 
pointing (“Show”), and/or asking a question that will prompt the child to verbally identify the object 
(“Say”). For example, a caregiver can ask, “See the ball?” Once it is clear that the child recognizes 
the object, she might say to the child, “Show me the ball.” Once the child has mastered this, she can 
ask the child, “What is this?” If the child is unable to do any one of these, the caregiver goes back 
one level. In this example, if she asks the child, “What is this?” and the child does not respond, the 
caregiver than would say, “Show me the ball,” and have the child point to the ball. The caregiver 
would provide positive reinforcement and then ask again, “What is this?” 

The 3N strategy (Notice, Nudge, Narrate) is a scaffolding strategy that also can be woven into the 
day. The strategy is designed to help the caregiver move the child from a current level of knowledge 
or skill to a higher level of competence. Using the 3N strategy, the caregiver can turn any activity into 
a learning experience for the child. The caregiver, observing the child, notices what the child is doing 
or preparing to do and uses words to describe it to the child or ask a question about it. Once the 
caregiver has noticed what a child is doing, she can then gently nudge the child toward a new learning 
opportunity. Finally she narrates, telling the story of what the child is doing to increase the child’s 
awareness of the significance of his or her own actions. For instance, “You chose the red ball!” adds 
color information and affirms the significance of the choice. 
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Supporting Materials 

In addition to a full set of games and the read-aloud books, providers were also given a family child 
care manual developed specifically for the intervention. The manual is approximately 20 pages and 
written in simple English and Spanish. It describes the LeamingGam.es approach, including the 
games, reading aloud, and the language priority strategies and provides a number of examples of 
each. 

The manual was accompanied by LeamingGames materials that could be posted by the caregiver and 
used for lesson plans and to track children’s progress. Providers also had a separate tracking 
document for each child, where they could record the games the child mastered as well as document 
read-alouds and enriched caregiving episodes. In addition, the providers were given sample lesson 
plans. (See Exhibit 3.2 for an example.) 

The Massachusetts LearningGames Professional Development 
Model 

Roles and Responsibilities for the LearningGames Implementation 

The professional development model that was designed for the Massachusetts study involved a 
system of training and support for three groups: two half-time study coordinators, would who 
supervise and supported the network home visitors; home visitors who would be trained in 
LearningGames and whose job was to provide technical assistance to providers; and the providers 
themselves. 

• The study coordinators, one of whom was bilingual, would provide support to the home 
visiting staff, primarily by accompanying them on one home visit every two months. 

They did not receive any specialized training or skill assessment but participated in an 
initial three-day training designed for the home visitors, which is described in the next 
bullet. In addition to the training, they met and communicated regularly with Dr. 
Sparling, the founder of MindNurture and the developer of LearningGames, and with one 
of two of his staff, who were trained technical assistance providers. They also 
accompanied Dr. Sparling and MindNurture staff on visits to family child care homes. 

• Home visitors would provide support to LearningGames providers. The home visitors 
were trained directly by Dr. Joseph Sparling and the MindNurture staff. The plan 
specified that home visitors would receive three days of training from Dr. Sparling and 
his staff in summer 2005 (before the intervention began), with a short refresher training 
in fall 2005. The home visitors would provide ongoing support to the providers in the 
form of feedback and technical assistance on LearningGames during hour-long, twice 
monthly visits to providers throughout the two-year training. 1 1 



1 1 The visitors also had non study-related responsibilities during the home visit, and continued to be 

responsible for visits to homes that were not part of the study, in addition to the LeamingGames homes. 
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Exhibit 3.2: Sample Weekly Lesson Plan 



This week with LearningGames Name: 

Date: 



Care Routines: 


Monday Tuesday Wednesday Thursday Friday 


Feeding 


Teach older children to say the wor” “spoo”.” Let the younger children just point to the spoon when 1 name it. 


Dressing 


Talk a lot about socks and shoes. Have the older children count socks and shoes. Have younger children point to shoes. 


Washing up 

Familiar Activities: 


Count fingers when we wash hands. Let the younger children hold up their fingers while 1 count them. 


Naming objects 


Name door and door knob Name door and door knob Name door and door knob Name window Name window 


Going for a walk 


Each day look for something green outdoors — name the things and talk about them 


Singing 


Sing” “Una boquita para com”r” at nap time. Sing” “Row, row, row your bo”t” when children are doing active play. 


Conversation plan 

Interactive Reading: 


Do back & forth babble talk at the diapering table. Make up rhyming words with the older children 


Using 3 S’s 


Make sure younger children see and show when 1 name the pictures. Let the older children say the names of many pictures. 


Books read: 

LearningGames: 


Yo! Yes? 1 Spy: Little Wheels My First ABC Alic’a’s Happy Day Finding Toys 


Infants 


#16. Ride-a-Horsie #16. Ride-a-Horsie #16. Ride-a-Horsie #16. Ride-a-Horsie #16. Ride-a-Horsie 


Ones 


#90. Making a Fun Path #90. Making a Fun Path #90. Making a Fun Path #90. Making a Fun Path #90. Making a Fun Path 


Twos 

Threes 

Fours 


#156. How about You? #156. How about You? #156. How about You? #156. How about You? #156. How about You? 


Other Activities: 


Make cookies today Use fingerpaints 



Source: Materials developed by Dr. Joseph Sparling, 2006. 



W 

CO 




• LearningGames providers would receive the predominance of training and support from 
the home visiting staff through the twice monthly home visits, and, in the second year of 
the intervention, from Dr. Sparling and MindNurture staff. Each of the family child care 
homes assigned to LearningGames would receive the materials described above, which 
were supplied by Dr. Sparling to the networks. They were expected to use the 
LearningGames approach to the best of their abilities with all children under 60 months 
of age. 

Ongoing supervision of the home visitors’ support for LearningGames was planned to be provided by 
two half-time project coordinators through study funding directed to them according to the request of 
the Massachusetts Office of Child Care Services (OCCS). Since the two coordinators had no prior 
experience with LearningGames , they were trained in summer and fall 2005 with the home visitors. 
The plan specified that the two coordinators would be supervised by a staff person from OCCS and 
advised by Dr. Sparling. The study coordinators were then expected to accompany each 
LearningGames home visitor on one visit every other month to a LearningGames home, for a total of 
six accompanied visits per year, and provide feedback to the home visitor about her work with family 
child care providers. One of the study coordinators was bilingual and could provide feedback and 
support to home visitors in Spanish when necessary. Study coordinators were to document for the 
study the providers who remained in the study (both in the treatment and control groups) and to verify 
that the LearningGames home visitors were not visiting and providing technical assistance to 
providers in the control group. Study coordinators also were expected to communicate this 
information, along with any general questions or concerns about the study, to the study team at Abt 
Associates. 

The implementation plan called for the home visitors to distribute the LearningGames materials to the 
providers and begin working with them immediately after completing the initial 3-day training. In the 
approximately hour-long visits to providers, home visitors were expected to cover several activities 
during each home visit. These included reviewing records, hearing about issues, and providing 
general feedback. The home visitor also would provide mentoring on specific LearningGames 
techniques. The training protocol instructed the home visitor to let the written game itself be the 
principal “teacher.” 

To make the game the “teacher,” the home visitor was instructed to read aloud the text from the game 
sheet. The provider was then to demonstrate the game and together, the home visitor and provider 
were to reflect on how the actual playing of the game was similar to or varied from the text. Home 
visitors were instructed to reflect on the positive aspects of the way the game was played and make 
suggestions about how it could be improved. Similarly, if there was time during the visit, they were 
to observe the caregiver providing interactive book reading and make suggestions about 
improvements in using the 3S strategies and other book reading techniques. The approach was for 
home visitors to observe and suggest but not to demonstrate, that is, not to actually do the games or 
other strategies directly with a child. 

Adaptations to the Model as Planned 

The adaptations to the professional development model involved changes in the schedule and 
intensity of training for both the LearningGames home visitors and the LearningGames providers. 
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Training and Support for LearningGames Home Visitors 

As we noted above, the professional development plan called for network home visitors to provide 
training and support to the family child care providers who were assigned to implement 
LearningGames. To develop the needed skill set to train providers on LearningGames, the initial 
plan called for home visitors to receive a 3-day training in summer 2005 and then immediately begin 
working with LearningGames providers. Then, after they had some practice as LearningGames 
coaches, they would participate in a short refresher training in the fall. As discussed in the section on 
Study Timeline in Chapter 2, the 3-day training for LearningGames had to occur as early as possible 
in summer 2005. However, this timing was not ideal for the study. Recruitment was not completed 
until spring 2005 and the home visitors were responsible for conducting the baseline observations of 
providers before implementation could begin. This data collection process was expected to last only 
a few weeks but ended up taking much of the su mm er for many of the networks. As a consequence, 
for many home visitors, several months elapsed between when they received the initial training and 
when they started working with providers. The actual implementation of LearningGames could not 
begin in many networks until fall/winter 2005, which represented a delay between their 
LearningGames training and their work with LearningGames family child care providers, creating 
potential issues with recall. 

While the initial plan was for Dr. Sparling to return in the fall and conduct a refresher training, it 
became clear early on that one additional training would be insufficient even without the initial delay. 
The developer determined that these two trainings were not enough to enable home visitors to master 
both LearningGames and home visiting protocols. Instead, after the initial training, Dr. Sparling held 
quarterly LearningGames training meetings with family child care network staff throughout the two 
years of the intervention. These meetings began in fall 2005 and went through summer 2007. In 
most cases, both the home visiting staff and their direct supervisors participated in the training 
sessions. 

Each of the quarterly training sessions were planned after discussions between Abt staff, project 
coordinators and Dr. Sparling to determine the agenda and address specific perceived needs of the 
home visitors and providers. For instance, a quarterly meeting in summer 2007 (approximately nine 
months after the implementation of LearningGames started) focused on giving home visitors tips on 
how to help providers enhance their enriched caregiving techniques. The home visitors were given 
short handouts that described the 3N and 3S techniques and then asked to work in pairs to come up 
with examples of how and when providers could use these techniques over the course of the day. 

They also did role plays in which one played the role of provider and one of home visitor. Finally, 
they watched videos of Massachusetts family child care providers who were in the LearningGames 
group and were asked to comment on what they observed in terms of enriched caregiving and what 
their approach as a home visitor would be to support the specific examples of such caregiving. 

In addition, starting midway through the first year of the intervention, Dr. Sparling and/or one of his 
two technical assistance staff directly visited the networks as well as some of the family child care 
homes in the study to provide direct technical assistance to home visitors, to see LearningGames in 
action, and to develop videotapes of providers that could be used to train home visitors on how 
LearningGames was being implemented. In the second year of the study, the Department of Early 
Education and Care (formerly OCCS) directed funding to MindNurture to enable Dr. Sparling and his 
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staff to provide a higher level of training and support than was possible with resources in the first 
year. 

Supporting Spanish-Speaking Home Visitors 

As will be described in Chapter 4, 41% of the family child care providers only spoke Spanish or were 
bilingual in Spanish and English (Exhibit 4.2). Networks assign all homes in which only Spanish is 
spoken and many English/Spanish homes to bilingual home visitors, some of whom requested 
additional language support because none of the LeamingGames professional development materials 
for home visitors were in Spanish. Further, Dr. Sparling and MindNurture staff did not speak 
Spanish. To help home visitors who requested language support, a designated table at quarterly 
training meetings was staffed by an Abt staff person who was bilingual. Elome visitors could choose 
to sit at that table and the Abt staff person would translate if necessary and/or check in with the home 
visitors to ensure that they understood the concepts being conveyed. In addition, many of the video- 
taped vignettes of LeamingGames in Massachusetts family child care homes that were created to 
facilitate home visitor training were from homes where Spanish was spoken. Finally, one of the study 
coordinators spoke Spanish and was able to provide feedback and support to home visitors when 
Spanish was preferred. 

Training for LeamingGames Providers 

The professional development plan called for one-on-one, in-person training of providers in visits to 
the homes by the LeamingGames home visitors. Some family child care networks reported that they 
added group training sessions for LeamingGames providers within their systems for home visitors to 
provide additional modeling of the games and techniques, going over the “rules” of the curriculum 
(e.g., 3S strategy and 3N strategy), and to answer questions about how to incorporate LeamingGames 
into their ongoing practices. In addition, Dr. Sparling and his staff provided direct training for 
providers in the second year of the study. All LeamingGames providers were offered opportunities to 
participate in group trainings and approximately 80% participated in at least one such training. 

Implementation of the Professional Development Model 

Fidelity of Implementation 

The study did not formally measure the fidelity of the professional development model. Evidence on 
fidelity comes from stakeholder interviews and a review of notes and logs. Together, these sources 
indicate that the professional development model was not fully implemented, in terms of the fidelity 
of the home visiting protocol, the adherence to twice monthly home visits, and the provision of 
technical assistance by project coordinators. 

Given the amount of new material to be mastered, both in terms of the LeamingGames curriculum 
and the home visit protocol, we expected that implementation of the professional development model 
might begin slowly and become stronger over time. While there were no formal assessments of 
fidelity of the home visits to the protocols, information provided by the project coordinators and 
MindNurture staff reinforced this expectation, as well as the belief that some of the home visitors 
needed additional support beyond the initial training and the quarterly training sessions. Project 
coordinators and MindNurture staff indicated that the home visitors with a firm understanding of the 
curriculum did well in transferring their knowledge to the providers and provided high-quality 
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technical assistance and mentoring. However, home visiting staff expressed different degrees of 
understanding of the curricula and some said that they needed an additional level of support, which 
occurred in Year 2 when the MindNurture staff were able to provide more direct technical assistance 
given the additional funding received. As one home visitor stated, “When support was there it was 
wonderful, but it wasn’t there from the beginning.” 

As described above, the original model called for project coordinators to shadow each of the home 
visitors on a LeamingGam.es visit approximately once every two months. The project coordinators 
estimated that they made 50-70% of the visits expected of them, in part due to scheduling issues and 
other job responsibilities of both the providers and home visitors. 

In addition, although training and materials clearly spelled out the expectation that home visits were 
to occur twice per month, anecdotal evidence indicates that these did not always occur, in part due to 
home visitor turnover and other issues described below. Interviews with family child care network 
staff indicate that, in some cases, providers only received one extended visit per month. In addition, 
some home visitors reported that, in some cases, they thought that the LearningGames homes did not 
require two visits a month for technical assistance and guidance. According to one home visitor, “I 
can call them and I know that I can trust they’ve done what’s on the chart [the lesson plan]. I don’t 
always go. Some are okay with less supervision.” 

Barriers to Implementation of the Professional Development Model 

Barriers to implementation fell into three categories: issues related to roles and responsibilities for the 
implementation, to the training of Learning Games providers, and to the train-the-trainer approach. 
Each set of issues is discussed below. 

Roles and Responsibility for Implementation 

Implementation of the planned roles and responsibilities for the implementation of LearningGames 
was an issue from the onset of the study. As expected, the Office of Child Care Services (OCCS) was 
involved in the planning and early implementation phases of the study; it recruited the family child 
care networks and also paid for the LearningGames materials and some of the time of Dr. Sparling 
and his staff. However, the amount of Dr. Sparling’s time originally supported by the state was 
insufficient for him to be more than a consultant to the project, as opposed to being in charge of 
LearningGames implementation. 

Similarly, while OCCS directed study funding to support the project coordinators, the coordinators 
were staff of child care resource and referral agencies, and did not report directly to either OCCS or 
MindNurture. During the study’s first few months, state staff held monthly conference calls with the 
project coordinators to monitor their activities; however this process was curtailed with state agency 
reorganization. When the study began, the position of OCCS commissioner was vacant and the 
agency was subsequently reorganized into a newly created Department of Early Education and Care 
(EEC), with a new commissioner starting in summer 2005. The new commissioner expressed interest 
in the study but determined that the state should not be directly involved in the administration or 
evaluation of a project that could influence the decision about adopting a curriculum for family child 
care homes statewide. While official state involvement was discontinued, the state did provide 
additional funding in Year 2 to Dr. Sparling and one of his technical assistance staff to support direct 
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training and other activities. The EEC Commissioner also sent a tha nk you letter to the providers in 
the study in recognition of their participation. While EEC staff did not provide any direct supervision 
to project coordinators, they continued to participate in the quarterly meetings to train home visitors. 
The reduced involvement of the state meant that Abt Associates was more involved with supporting 
implementation on a day-to-day basis than originally intended. 

Initially, state staff held conference calls with the project coordinators to learn about issues with the 
implementation of LeamingGam.es at specific networks and/or in specific homes and to monitor their 
work. For several months after the state decision not to directly oversee the implementation, Abt 
Associates held these calls. In the last year of the study, this became the direct role of Dr. Sparling 
and staff. 

Training the LearningGames Providers 

Home visitors initially had to address several provider perceptions about LearningGames and about 
the home visits that threatened the full implementation of the model. First, home visitors reported 
anecdotally that providers felt that the LearningGames activities, such as showing a baby her face in a 
mirror or singing to children at mealtimes, were what they were “already doing” despite the home 
visitors’ assessment that they were not. Therefore, in order for home visitors to convince providers 
that using LearningGames could fundamentally change their interactions with children, home visitors 
required a strong grasp not only of the specific games but also of the developmental theory 
underlying the games. Home visitors also needed enthusiasm for the LearningGames approach, a 
solid background in early childhood development, and a strong skill set for working with family child 
care providers to help them developmentally enhance care. 

A second provider perception that needed to be addressed by home visitors dealt with the value of 
working with one or two children and the possibility of this happening in homes. Overall, family 
child care network staff reported that providers were skeptical about the possibility of conducting 
LearningGames with individual children in their homes. Providers perceived that there were too 
many demands on their time and that group learning, despite the age differences of children, and free 
play were the only feasible approaches. Therefore, many of the initial quarterly training sessions 
focused on giving providers tips on how to focus attention on one or two children at a time. By the 
end of the two-year period, home visiting staff reported that participating providers thought an 
individualized approach feasible. 

Some providers also reported issues with the LearningGames home visits. First, although there were 
a substantial number of technical assistance topics to be covered in one hour, some providers voiced 
discomfort at being coached and observed for that length of time. Second, network staff reported that 
some providers felt as if they were being judged or scrutinized, particularly during those visits when 
project coordinators accompanied the home visitors. Finally, the LearningGames professional 
protocol called for the provider and home visitor to read the game aloud. This practice was designed 
so that reviewing the game would be shared and the home visitor would not be put into the position of 
“instructor.” However, network staff reported that reading aloud was not always comfortable, as 
neither party had experience doing so with other adults and were not always confident enough in their 
reading aloud skills to do so. 
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Finally, while it appeared that many providers found their home visitors helpful, they also wanted 
direct training from the developer. Direct training of the family child care providers was considered 
but then dismissed because it did not seem feasible, since homes were located throughout the state. 

On one hand, it did not seem possible for MindNuture to schedule and conduct up to 23 separate 
group trainings for each of the participating networks or regional offices. The other option was to 
offer one or a small number of centralized trainings. Given the long working hours of family child 
care providers and the need for them to travel to a centralized locations for this option, it seemed 
unlikely that many providers would be able to participate. However, network staff reported that 
providers wanted direct training, which was echoed by providers themselves in the second year of the 
study when they received it. (One of the reasons why it became feasible in the second year was that 
the high level of attrition reduced the number of LearningGames providers, making coordination of 
group trainings much more feasible.) Network leadership, state staff and the developer indicated that 
they believed that direct training earlier in the intervention would have been more helpful than relying 
completely on home visitors, as providers would be receiving training directly from the 
LearningGames expert, instead of from home visiting staff, many of whom had not yet mastered the 
approach. 

Train-the-Trainer Approach With Home Visiting Staff 

There were a number of issues that impeded the LearningGames home visitors from being optimally 
effective in providing mentoring and support to the LearningGames homes. These issues included 
time needed for training and mastery of the LearningGames approach, caseloads, staff turnover, and 
less than optimal initial support. Each of these issues is discussed below. 

Insufficient time for mastery of LearningGames concepts . As stated earlier, the original plan was for 
a total of 5 days of training (3 initial days of training and two 1-day refresher trainings). However, 
home visitors were asked to attend 10 days of training (an initial 3-day training and 7 quarterly day- 
long meetings) over the course of the two years. Home visitors needed additional time beyond the 
training sessions to develop their understanding of LearningGames. 

At least initially, the professional development model enabled home visitors to stay just one step 
ahead of providers. After receiving the summer 3-day training on the LearningGames approach and 
the home visiting protocol, home visitors were to read and master specific LearningGames and help 
the providers choose one to do with each of their children. Prior to every LearningGames visit, they 
needed to review and understand the LearningGames that were being played with each child in the 
home and to learn the next games to be played. They needed to know the games well enough to be 
able to suggest ways in which the games could be deepened and broadened and to be able to explain 
the developmental significance of these activities to the providers. Since a specific game is selected 
for each individual child, and the providers’ homes had children who varied in age, this could mean 
home visitors needed to learn many new games to stay on top of things during any given week. 

While knowledge of specific games eventually would accumulate, at the outset there was a lot of 
material to go through. In addition, to provide general advice and support, home visitors also needed 
to be able to identify and support instances when the caregivers used LearningGames learning 
strategies during the course of the day (enriched caregiving, using 3S and 3N) or to identify 
opportunities where caregivers could use these strategies. 
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In addition to receiving training and mastering LearningGames, home visitors were required to 
document each of the home visits with the LearningGames providers. They were also asked to 
periodically fill in fidelity measures that indicated how well the providers were implementing each of 
the elements of LearningGames. Finally, home visitors were asked to coordinate with project staff 
who would accompany them on a visit every other month. 

High home visitor caseloads . Initially, LearningGames home visitors were assigned to provide 
mentoring to three to ten providers doing LearningGames, with about one-third of home visitors 
working with five or more LearningGames providers. The number assigned depended upon the 
number of homes within a network participating in the study and the number of home visitors that the 
network decided to train. By su mm er 2007, home visitors were working with between two and seven 
LearningGames providers. 

Home visitors who supported LearningGames homes also supported additional providers in the 
network, although none of these providers belonged to the control group. Home visitors’ overall 
caseloads, that is, the total number of providers that they were required to visit, had a potential impact 
on the time they needed to do the LearningGames intervention. As described in Chapter 2, to 
participate in the study, networks had to agree that their home visitors would be able to make two 
LearningGames visits per month to the LearningGames homes. They did not, however, have to 
guarantee that the home visitors had a specific caseload of providers that would permit them to 
implement the model as planned. As described in Exhibit 2.2, the participating networks reported 
that the average home visiting caseload was 20 and approximately one-third of the networks reported 
case loads of 15 or fewer providers. However, anecdotal information indicated that, in reality, for 
many of the home visitors the actual caseload was higher, especially when visits were factored in for 
nutritional monitoring visits for the Child and Adult Care Food Program and/or visits to homes with 
children who were receiving child care because they were in child protective services, which included 
some of the other responsibilities they had to perform. In some cases, the network supervising staff 
of the home visitors augmented support to the LearningGames providers by conducting some home 
visits themselves. 

Home visitor turnover . Approximately 25 home visitors initially were assigned to provide 
LearningGames support. At the end of the study, 1 1 of these home visitors were no longer in that 
role; approximately two-thirds of them had changed roles in the network and the rest had left their 
agencies. 12 In Chapter 2, we noted that because they were selected by networks as opposed to 
randomly selected, LearningGames home visitors could be different at baseline than control home 
visitors and this could contribute to any differences in provider outcomes. However, the high level of 
turnover of LearningGames home visitors might, in fact, reduce the chances that the selected 
LearningGames home visitors were systematically different than their control counterparts. 

There were no formal provisions for training new home visitors. If the former LearningGames home 
visitor remained with the network, she generally provided training to the new home visitor. In all 
networks, the home visitor supervisor attended the quarterly training sessions and was familiar with 
LearningGames and able to offer informal support, as did home visitors who changed roles but stayed 



12 Because random assignment occurred at the provider level, no homes were lost from the study as a result of 
home visitor turnover. 
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employed in the network. The new home visitor attended the quarterly meetings, but the next 
scheduled meeting may have occurred up to three months after she assumed her LeamingGames 
duties. In Year 2, MindNurture technical assistance staff made special attempts to meet and train the 
new home visitors. In interviews, staff from family child care networks reported that retraining went 
more smoothly for those systems where home visitor supervisors also participated in quarterly 
meetings and were familiar with LeamingGames as they could provide the new home visitors with 
training and support. Retraining of new home visitors was not documented. 

We do not know if later implementation was stronger or weaker because of home visitor turnover. 
While new home visitors may not have been as able to support LeamingGames as their counterparts, 
LeamingGames providers likely gained more facility with the model over time and may have not 
needed as much support. In addition, as time went by, home visitor supervisors who participated in 
quarterly meetings were more knowledgeable about LeamingGames and able to augment training and 
support provided by project coordinators and MindNurture staff. It is unclear which set of factors 
was stronger: home visitor turnover versus improvements over time in other aspects of the 
professional development model and in the providers’ implementation of LeamingGames. 

Training and experience of home visiting staff. Interviews with network and other staff indicated that 
the level of education and experience of home visitor made a difference in the amount of training and 
support that was needed to give them the tools and skills to mentor the LeamingGames family child 
care homes. As described earlier, the education attainment of the home visitors varied greatly, with 
approximately 16% with a high school degree, 27% with a bachelor’s degree, and most of the 
remaining either having an associate’s degree or some college. It was perceived that additional 
support, beyond the regional training sessions, was needed for those with less experience. To provide 
additional support MindNurture staff made visits to the family child care networks and provided 
additional coaching and guidance for home visiting staff and their supervisors in the second year of 
the intervention. 

Support by project coordinators and MindNurture staff . As stated earlier, the project coordinators 
received training at the same time as the LeamingGames home visitors. They were expected to 
provide guidance and technical assistance to home visitors, even though they had the same level of 
training, at least initially. Respondents to interviews in summer 2007 indicate that more training in 
LeamingGames as well as in their role as project coordinators might have been helpful. 

Providers’ Implementation of the Learn/ngGames Program Model 

To understand how well the LeamingGames homes implemented the approach, we drew from (1) 
interviews with state staff, MindNurture staff, project coordinators and network staff; (2) a home 
visitor fidelity rating checklist completed in spring 2006 (approximately 18 months after 
implementation); and (3) a fidelity measure created from a subset of items taken from the fall 2006 
provider observations. Each is described more fully below. 
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Perceptions of Staff of MindNurture and Family Child Care Networks of Providers’ 
Implementation of LeamingGames 

The LeamingGames developer, his staff and network home visitors reported similarly that there was 
substantial variation in the degree to which providers understood and were able to implement 
LeamingGames. Some of the home visitors reported that they believed some of their providers had 
integrated the LeamingGames approach throughout their daily activities, while others did the 
LeamingGames activities for “show” during home visits but did not do them consistently throughout 
each day. Because implementing LeamingGames fully and accurately was a relatively large 
commitment, they believed that those providers who were really serious, motivated and had the 
necessary basic skills were those who were most successful. 

Network staff reported that many of the providers liked LeamingGames but some did not. For 
instance, at one site network staff reported that providers were enjoying implementing the program 
and none of them wanted to stop using it. When providers criticized the approach, it was generally 
for two different reasons. On the one hand, network staff said that some providers felt that 
implementing LeamingGames was too labor intensive and too complicated since it required delivery 
over the entire course of the day and individualized interactions with each child. Other providers, 
however, thought that the content of the approach was overly simple and that it did not add anything 
to the way they already interacted with children. 

Family child care network staff themselves generally reported that they had positive opinions about 
LeamingGames but had mixed opinions about for which providers it was most appropriate. Many 
staff who were interviewed reported that the LeamingGames curriculum “works” if used as it as 
intended. Flome visitors from several networks reported that the overall approach was simple and 
that the games were valuable in that they are easy to implement, age appropriate and complete. 

While all of the network staff who were positive about the curriculum seemed to believe that it was 
an appropriate tool for new providers, there were mixed opinions about its usefulness for more 
experienced providers. Some home visitors considered LeamingGames a valuable tool for seasoned 
providers because it reinforced their belief that what they are doing is meaningful, the importance of a 
one-on-one, individualized approach, and the importance of weaving learning opportunities 
throughout the day. Flowever, other home visitors felt that because the curriculum seemed simplistic 
to many experienced providers it was less appropriate for them. 

Fidelity of Implementation of LeamingGames 

In spring 2006 home visitors were asked to rate the level of fidelity of LeamingGames being 
implemented by family child care providers. Home visitors submitted ratings for approximately 70% 
of the providers who were in the study at that time. Despite indicating during stakeholder interviews 
that the level of implementation was mixed, home visitors rated almost all of the family child care 
providers as doing all of the LeamingGames elements “often” or “always.” It must be kept in mind 
that these high ratings were given by technical assistance staff; the ratings of objective observers in 
other studies are frequently lower (Judkins et al., 2008). These ratings included playing 
LeamingGames, doing interactive book reading, and enriched caregiving. (See Exhibit 3.3.) 
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Exhibit 3.3: Home Visitor Rating of Fidelity of Implementation by Providers (n=67 providers) 





Never 


Some- 

times 


Often 


Always 


1. Makes weekly plans using the form 
LearningGames This Week? 


5% 


10% 


18% 


67% 


2. Shares weekly plans with you? 


3% 


18% 


17% 


62% 


3. Does Interactive Book Reading every day with 
every child? 


0% 


6% 


21% 


73% 


4. Plays LearningGames activities every day with 
every child? 


0 


6% 


32% 


62% 


5. Discusses LearningGames and Interactive Book 
Reading with you and demonstrates games and book 
reading when you visit? 


0 


8% 


27% 


66% 


6. Sends LearningGames activities (1 or 2 at time) 
home to parents when they are used? 


2% 


20% 


16% 


63% 


7. Sends conversation books home to parents; tells 
parents how important book reading is? 


7% 


20% 


18% 


56% 


8. Uses Enriched Caregiving? 


0% 


5% 


12% 


83% 


9. Uses 3S strategy (See/Show/Say) to get responses 
from children during Interactive Book Reading? 


0% 


8% 


8% 


85% 


10. Uses 3N strategy (Notice/Nudge/Narrate) during 
free play and enriched caregiving? 


0 


9% 


8% 


83% 



Measurement of Fidelity in LearningGames and Comparison Homes 

In addition to relying on the home visitors’ ratings of family child care homes, Abt Associates created 
a fidelity rating scale using those items from the provider observation that were most closely aligned 
with the LearningGames approach. Exhibit 3.4 describes each of these variables, which largely focus 
on extended and enriched interactions between caregivers and children throughout the day. As the list 
shows, it would be possible for the control group also to be practicing these behaviors as they are 
consistent with high-quality caregiver-child interactions. For each of the items, we developed a 3- 
point scale, with a rating of “3” indicating that providers were engaging in the activity at a level that 
would be considered “fully implementing” the LearningGames approach. 
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Exhibit 3.4: Elements of the LearningGames Fidelity Scale 



Variable 


Rating 


Extended verbal/non-verbal interactions with 
individual or pairs of children (Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


Nudges children to try something new (Source: 
QUEST item) 


1: never or infrequently; 2: occasionally; 
3: often/consistently 


Enriches routine through language 
interactions/learning (Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


Language-rich interactions (Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


Encourages children to engage with print (Source: 
QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


If child < 12 months'. Encourages infants to 
explore/be active (Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


If child > 36 months in care'. Helps children talk about 
what they are doing/thinking through open-ended 
questions (Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


If child > 36 months in care\ Extended rich 
conversations with individual or pairs of children 
(Source: QUEST item) 


1 : never or infrequently; 2: occasionally; 
3: often/consistently 


Proportion of time in meaningful talk with individual 
children (extended conversation; singing/back-and- 
forth verbal games with infants or toddlers) (Source: 
TALK) 


1 : > 5% 2: 5-25%; 3: 25% or more 


Proportion of time in routine activities that provider is 
playing, demonstrating/discussing with children 
(Source: SNAP) 


1 : 0%; 2: 1-25%; 3: 25% or more 


Proportion of reading aloud that is with individual or 
pairs of children (Source: RAP) 


1 : 0%; 2: 1-75%; 3: 76% or more 



The total number of possible points on the fidelity scale ranged from 1 1 to 33. 13 There was a 
significant difference on the fidelity score between the LearningGames providers and the control 
providers (Exhibit 3.5). The average fidelity score for the LearningGames providers was 18.4, 
compared with an average score of 15.3 for the control providers. In addition, we created a score 
based on the proportion of items on which a provider received a rating of “1,” meaning that they 
never or infrequently exhibited the behavior; therefore receiving the lowest rating on multiple items 
was seen as an indicator of low fidelity to the LearningGames approach. The average proportion of 

13 For the items that depended on the ages of children in the home, missing values were set to the mean score 
for that home on the other items. 
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items with a rating of “1” was 50% for the LearningGame s providers and 69% for the control 
providers, which was significantly different. This suggests that the LeamingGames training was 
effective at changing the behavior of the family child care providers in line with the LearningGame s 
objectives. 



Exhibit 3.5: Scores on Fidelity Scale after Two Years of LeamingGames Intervention by 


Treatment Status 










Treatment 

Providers 




Statistical 




Control Providers 


Significance of 
Difference 


Measure 


Mean 


Mean 


p-value 


Fidelity rating (out of 33) 


18.4 


15.3 


.03 


Proportion of fidelity items 
where provider scored as “1” 


50% 


69% 


.0001 
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Chapter 4: Impacts of LearningGames on Family 
Child Care Providers 



Overview 

In this chapter, we present findings on the impact of LearningGcimes on provider practices at two 
points during the intervention: after one year of LearningGcimes and after two years, which 
represents the end of the study. In this chapter, we describe the analytic sample, the outcomes tested, 
strategies for developing the outcome constructs, and the analytic approach to estimating impacts. To 
summarize, the analyses indicate that by the end of two years of intervention, LeamingGam.es 
showed significant impacts on provider behavior, although these same impacts were not evident after 
only one year of implementation. 

Research Questions 

As described in Chapter 1 , LearningGames is hypothesized to improve the developmental outcomes 
for the children whose providers are trained on the approach. Changes in providers in 
LearningGames homes are seen as intermediate outcomes that are necessary precursors to impacts on 
children. The study investigated impacts on providers at the end of one year of the LearningGames 
intervention (intermediate impacts) and at the end of two years (posttest impacts). At the end of one 
year, the implementation of LearningGames was halfway through its planned intervention period. 
Analyses investigated the evidence that the first step in the logic model (changes in providers) was 
occurring by examining a relatively broad set of outcomes that reflected high-quality practices for 
supporting children’s learning and development, which were aligned with but not confined to the 
objectives of the LearningGames program. The two-year data collection and analyses represented the 
full LearningGames treatment. To avoid statistical complexities associated with testing a large 
number of outcomes, the two-year impact analyses were limited to a small set of provider outcomes 
that measured key goals of Learn ingGames. 

Providers at Baseline 

The results of the impact analyses are preceded by discussion of the providers at baseline. Even 
when, as is the case in the current study, the integrity of the random assignment is preserved, the 
resulting samples may not be equivalent on all baseline characteristics. These differences do not 
threaten the internal validity of the study but should be adjusted for in any subsequent impact 
analyses. 

Interactions with Children 

Exhibit 4.1 presents the scores for the treatment and control providers on the baseline observations, 
which were conducted by the child care network home visitors prior to the LearningGames training. 
There were no significant differences between the LearningGames and the control providers on either 
the QUEST Caregiver Rating Scale or the Arnett CIS. On the QUEST, which uses a 3-point Likert 
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scale, treatment and control providers were rated near the top of the rating scale in all areas of 
practice with children. Similarly, on the Arnett, which uses a 4-point Likert scale, both treatment and 
control providers were rated near the top of the scale on their responsiveness, warmth and 
engagement with children. Note that the uniformly high ratings of providers may be at least in part a 
function of having agency staff conduct the observations. These staff did not undergo rigorous 
training to a level of research reliability, and their role in providing technical assistance and training 
for the providers may have provided pressure to present as positive a picture as possible through their 
ratings. There is no reason to believe that agency staff rated LeamingGam.es providers differently 
than providers in the control group and therefore any “error” rates would be evenly distributed among 
the two groups. However, overly inflated ratings may have a ceiling effect on variation between 
providers in the treatment and control groups. 



Exhibit 4.1 : Baseline Scores for Providers on QUEST Caregiver Rating Scale and Arnett 
Caregiver Interaction Scale by Treatment Status 




T reatment 
Providers 


Control 

Providers 


Statistical 
Significance 
of Difference 


Subscore (# items) 


Mean 


Mean 


p- value 


QUEST Caregiver Rating Scale (1- 3) 


Caring and responding (10) 


2.78 


2.78 


.90 


Using positive guidance and discipline (9) 


2.63 


2.63 


.99 


Supervision (4) 


2.83 


2.84 


.84 


Does no harm (5) 


2.92 


2.99 


.48 


Supporting social emotional development (8) 


2.69 


2.72 


.43 


Supporting play (8) 


2.74 


2.76 


.56 


Instructional style (5) 


2.53 


2.58 


.35 


Learning activities and opportunities (11) 


2.53 


2.59 


.308 


Supporting language and literacy (11) 


2.53 


2.53 


.98 


Television and computers (2) 


2.76 


2.76 


.97 


Arnett Caregiver Interaction Scale (1-4) 


Overall rating (26) 


3.46 


3.49 


.49 


Sample size 


153 


153 


316 



Demographic Characteristics 

Exhibit 4.2 presents the characteristics of the LeamingGames and the control providers and homes at 
baseline. Almost 70% of the providers in the analytic sample were between 36 and 50 years of age. 
On average, they were experienced as caregivers and had operated family child care providers for 
more than five years. Nearly half of the providers had some college education (46%), although only 
14% had a four-year college degree. About one third of providers reported having a CDA. Over a 
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third of the providers were white non-Hispanic and 45% were Hispanic, with a much smaller number 
of Black providers. Just over half of the providers spoke English primarily or exclusively in their 
family child care homes (58%). 



Exhibit 4.2: Baseline Demographic Characteristics of Providers by Treatment Status 




Control 

Group 

% 


Treatment 

Group 

% 


Total 

Sample 

% 


Statistical 

Significant 

of 

Difference 

p-value 


Age at assignment 










21 years old or younger 


0 


0 


0 




22 - 35 years old 


7.3 


9.2 


8.3 




36 - 50 years old 


74.5 


63.0 


68.8 




51 years old or older 


18.2 


27.8 


22.9 




Average age in years 


36.6 


40.3 


38.4 


.35* 


Background/ethnicity 








,21 b ' c 


White, non-Hispanic 


44.6 


29. 1 b 


36.9 




Hispanic 


41.1 


49.1 


45.0 




Black 


10.7 


10.9 


10.8 




Other 


3.6 


10.9 


7.2 




Language(s) used with children in care 








,62 b 


Primarily English 


55.4 


61.1 


58.2 




Primarily Spanish 


14.3 


11.1 


12.7 




Bilingual (English/Spanish) 


26.8 


25.9 


26.4 




Bilingual (English/Other) 


3.6 


1.9 


2.7 




Highest level of education 








.3t 


Less than high school 


14.3 


10.7 


12.5 




High school diploma 


37.5 


44.6 


41.1 




Some college, no degree 


4.7 


5.3 


4.9 




AA degree 


31.2 


28.6 


29.9 




BA degree 


14.3 


10.8 


11.6 




Child Development Associate Certification 










CDA 


35.7 


29.0 


32.4 


.22? 


Years in Family Child Care 








. 13b a 


Less than 5 years 


17.9 


26.6 


20.7 




More than 5 years 


82.1 


76.4 


79.3 




Sample size 


144 


147 


297 




a Significance based on t-test b Significance based on chi-square test. 

c Difference in proportion of LeamingGames and control providers who were white, non-Hispanic vs. other was not 
significant, (p = .12). 
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Provider Impacts After One Year of Implementation 



As described in Chapter 3, the implementation process in the first year of the intervention appeared to 
be less intensive and systematic than planned. The observations at the end of the first year provided 
an objective test of whether or not the uneven implementation support in the first year resulted in 
significant changes in the practices of the LeamingGames providers. As described in Chapter 2, at 
the time of the one-year observations, 203 providers (57%) from the originally assigned sample of 
353 providers remained in the study, which included 98 LeamingGames providers and 105 control 
providers. 

Baseline Differences in the Analytic Sample 

The first question tested was whether the samples of treatment and control providers who remained in 
the sample after one year were statistically different from each other, on either demographic 
characteristics or interactions with children. There also were no significant differences between the 
treatment and control providers in the analytic sample on demographic characteristics (Exhibit 4.3). 
Similarly, there were no significant differences between the LeamingGames and the control providers 
in the analytic sample after one year of implementation on the QUEST Caregiver Rating Scale and 
the Arnett CIS, which were administered at baseline by staff from the child care networks (Exhibit 
4.4). On the QUEST, which uses a 3-point Likert scale, treatment and control providers were rated 
near the top of the rating scale in all areas of practice with children. Similarly, on the Arnett, which 
uses a 4-point Likert scale, both treatment and control providers were rated near the top of the scale 
on their responsiveness and engagement with children. 



Exhibit 4.3: Selected Baseline Demographic Characteristics of Analytic Sample after One 
Year of LeamingGames Intervention by Treatment Status 





Control 

Group 

% 


T reatment 
Group 

% 


Total 

Sample 

% 


Statistical 
Significant of 
Difference 

p-value 


Providers are white, non-Hispanic 


43.2 


30. 3 b 


35.8 


.11 


Provider speaks primarily English with 
children 


51.4 


59.3 


58.2 


.44 


Provider has college degree (AA or higher) 


42.6 


40.7 


41.9 


.31 


Sample size 


90 


101 


191 
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Exhibit 4.4: Baseline Scores on QUEST Caregiver Rating Scale and Arnett Caregiver 
Interaction Scale for Analytic Sample after One Year of LeamingGames Intervention by 
T reatment Status 





Treatment 

Providers 


Control 

Providers 


Statistical 
Significance 
of Difference 


Subscore (# items) 


Mean 


Mean 


p-value 


QUEST Caregiver Rating Scale (1- 3) 








Caring and responding (10) 


2.7 


2.8 


.45 


Using positive guidance and discipline (9) 


2.6 


2.6 


.53 


Supervision (4) 


2.8 


2.8 


.87 


Does no harm (5) 


2.9 


2.9 


.62 


Supporting social emotional development (8) 


2.7 


2.7 


.89 


Supporting play (8) 


2.7 


2.7 


.95 


Instructional style (5) 


2.5 


2.6 


.37 


Learning activities and opportunities (11) 


2.5 


2.5 


.85 


Supporting language and literacy (11) 


2.5 


2.5 


.94 


Television and computers (2) 


2.7 


2.8 


.44 


Arnett Caregiver Interaction Scale (1-4) 








Overall rating (26) 


3.4 


3.5 


.49 


Sample size 


98 


105 


203 



Impacts on Providers 

The analyses of the one-year provider outcomes were considered exploratory, since the study design 
called for the primary test of impacts on providers to be based on outcomes at the end of the full two 
years of the intervention. Differences in the behavior of the treatment and control providers were 
evaluated based on five observation measures that were administered by independent observers at the 
end of one year of LeamingGames. The measures (all described in Chapter 2) included: the QUEST 
Caregiver Rating Scale, the Arnett CIS, adapted versions of the OMLIT Snapshot of Activities and 
the OMLIT Read Aloud Profile, and a draft time-sampled measure — Providers’ Interactions with 
Children (PIC) — which was developed for the current study to assess the extent of LeamingGames 
implementation (or LeamingGames -] ike activities). 

From these five instruments, 18 outcomes were constructed to assess the impact of LeamingGames 
on family child care providers’ instructional approaches and caregiving activities. Some of the 
outcomes were designed to assess instructional strategies that were specifically targeted by 
LeamingGames, particularly those associated with enriched caregiving and interactive book reading. 
Other outcomes represented high-quality practices to support children’s learning and development. 
Together, the outcomes provided a broad portrait of the types of activities, interactions, and 
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instructional approaches that providers use in family child care homes. Note that this set of outcomes 
developed for the one-year impacts is different from the three composite outcomes used in the two- 
year impact analyses. This is partially because of the differences in the measure battery but more 
importantly because we wanted the primary impact analyses to be based on a small set of outcomes, 
to reduce concerns associated with multiple comparisons. (The description of the composite 
outcomes developed for the two-year analyses is in “Provider Outcome Measures” in the section 
below on Provider Impacts after Two Years of Intervention.) 



Exhibit 4.5: Impacts on Provider Practices after One Year of LeamingGames Intervention 




Treatment 

Providers 


Control 

Providers 


Statistical 

Significance 

of 

Difference 




% 


% 


p-value 


OMLIT Snapshot of Activities 








Children involved in high-value activities 3 


50.2 


53.2 


.15 


Provider highly involved in children’s activities (instructing, 
reading, discussion) 


31.0 


32.3 


.65 


Provider not involved in children’s activities 


33.0 


35.0 


.39 


Provider Interactions with Children 








Enriched caregiving with one or two children 


35.9 


28.8 


.04* 


QUEST Caregiver Rating Scale (1- 3) 


Mean 


Mean 




Provider support for cognitive, language, and social development 


2.09 


2.12 


.57 


Arnett Caregiver Interaction Scale (1-4) 


Mean 


Mean 




Responsive 


3.36 


3.28 


.35 


Warm 


3.76 


3.70 


.41 


Attached/Engaged 


3.63 


3.57 


.41 


Permissive 


3.05 


3.03 


.84 


OMLIT Read Aloud Profile 


% 


% 




Reads aloud to one or two children 


34.7 


32.4 


.73 


Uses “see, show, say with one or two children 


31.6 


29.5 


.74 


Points out features of print 


90.8 


87.6 


.46 


Points out sounds/letters or sound-letter link 


7.1 


12.4 


.21 


Promotes print motivation 


75.5 


77.1 


.78 


Introduces/highlights vocabulary 


25.5 


25.7 


.97 


Supports comprehension: provides information 


90.8 


85.7 


.26 


Supports comprehension: links to children’s experience 


31.6 


27.6 


.53 


Supports higher order thinking through the use of questions 


12.2 


12.4 


.98 


Sample size 


98 


105 




1 “Includes reading and literacy activities; dramatic, creative, sensory and fine motor play, blocks, and games. 


Key: * = p < .05; ** = p< .01. 
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There was a statistically significant difference between the LearningGames and control providers on 
only 1 of the 18 provider outcomes tested: proportion of time that providers used “enriched 
caregiving” (Exhibit 4.5). This outcome described the provider’s introduction of cognitively 
stimulating language play and interactions during caregiving routines such as washing hands and 
eating snack; narrating, talking about, giving feedback on, or asking questions about what children are 
doing; and nudging children to try something new or to extend an activity by themselves. It should 
be noted that testing this number of outcomes means that by chance alone, one or two of the contrasts 
may be significant. 

Provider Impacts After Two Years of Implementation 

At the time of the two-year observations, 150 providers from the originally assigned sample of 353 
providers remained in the study, 69 LearningGames providers and 81 control providers. 

Baseline Differences in the Analytic Sample 

As with the Year 1 sample, for the observation measures administered at baseline, the 
LearningGames and control providers in the two-year observation sample did not differ at a 
statistically significant level on any of the baseline scores for the QUEST or the Arnett CIS (Exhibit 
4.6). 
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Exhibit 4.6: Baseline Scores on QUEST Caregiver Rating Scale and Arnett Caregiver 
Interaction Scale for Analytic Sample after Two Years of LearningGames Intervention by 
Treatment Status 





T reatment 
Providers 


Control 

Providers 


Statistical 
Significance 
of Difference 


Subscore (# items) 


Mean 


Mean 


p-value 


QUEST Caregiver Rating Scale (1- 3) 








Caring and responding (10) 


2.8 


2.8 


.75 


Using positive guidance and discipline (9) 


2.6 


2.6 


.83 


Supervision (4) 


2.9 


2.8 


.27 


Does no harm (5) 


2.9 


2.9 


.82 


Supporting social emotional development (8) 


2.7 


2.7 


.89 


Supporting play (8) 


2.8 


2.8 


.96 


Instructional style (5) 


2.5 


2.6 


.37 


Learning activities and opportunities (11) 


2.5 


2.5 


.88 


Supporting language and literacy (11) 


2.5 


2.5 


.93 


Television and computers (2) 


2.7 


2.8 


.31 


Arnett Caregiver Interaction Scale (1-4) 








Overall rating (26) 


3.5 


3.5 


.98 


Sample size 


69 


91 


150 



For selected demographic variables, the LearningGames and control providers remaining in the 
analytic sample were not statistically different (Exhibit 4.7). 



Exhibit 4.7: Selected Baseline Demographic Characteristics of Analytic Sample after Two 
Years of LearningGames Intervention by Treatment Status 




Control 

Group 


Treatment 

Group 


Total 

Sample 


Statistical 
Significant 
of Difference 




% 


% 


% 


p-value 


Providers are white, non-Hispanic 


42.4 


31. 6 b 


35.7 


.13 


Providers speak primarily English with children 


56.1 


60.7 


58.9 


.52 


Providers have college degree (AA and higher) 


42.1 


38.3 


40.0 


.78 


Sample size 


56 


55 


111 
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Characteristics of Homes in the Two-Year Analyses 



Based on the observations, we could characterize the homes in the final impact analyses. As shown 
in Exhibit 4.8, the average enrollment of these homes was between 5 and 6 children, although 40% of 
the homes were larger (between 6 and 8 children). As noted earlier, the eligibility requirements at the 
beginning of the study included the requirement that the home had at least one child in care under 3 
years of age. Additionally, it was desirable that there be at least one infant in care (defined as less 
than 12 months of age). This second criterion was intended to increase the chances that, at the end of 
the intervention, there would be children in care who were at least 3 years of age, which was the 
target age for the intended assessments. It was understood that this requirement could not guarantee 
that a sample home would have a preschool child at the end of the intervention, since, over the two 
years of the intervention the children who were there at the time of random assignment could leave 
the home and be replaced, resulting in homes with younger children. In fact, over 85% had at least 
one preschool-age child in their home, while only 45% had at least one infant age child. 

The modal home in the sample contained both toddlers (children between 1 and 3 years of age) and 
preschoolers (children 3 to 5 years of age). Just over a third of homes contained infants, toddlers and 
preschool children. (A number of homes also cared for school-age children in the hours after school, 
although these children were not the focal age for the intervention and were not present in the 
observations.) Only a quarter of the providers cared for a related child, defined as either their own 
child or another related child. In a large number of homes, all of the children were from English 
language backgrounds. However, a significant number of homes served a mix of English-language 
and bilingual children. In the majority of homes with at least one English Language Learner, the 
provider spoke the home language(s) of all of the children in care. Less than 20% of the homes in the 
sample were accredited. 

Treatment and control homes differed only on the match between the provider’s language and the 
children’s language. There was more likely to be a match between the provider’s language and the 
children in her care in the control homes. 



Exhibit 4.8: Characteristics of Child Care Homes in the Analytic Sample after Two Years of 
LeamingGames Intervention by Treatment Status 




Control 

Group 


Treatment 

Group 


Total Sample 


Statistical 
Significance 
of Difference 




% 


% 


% 


p-value 


Size of homes: # of children enrolled 3 


% homes with < 3 children 


13.6 


11.6 


12.7 


.80 


% homes with 3-5 children 


68.1 


65.2 


66.7 


.95 


% homes with 6-8 children 


14.6 


23.2 


19.6 


.12 


% homes with > 9 children 


3.7 


0.0 


2.0 


.47 


Ages of children enrolled in homes 


% of children 0-11 months 


22.3 


24.4 


23.3 


.91 
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Exhibit 4.8: Characteristics of Child Care Homes in the Analytic Sample after Two Years of 
LeamingGames Intervention by Treatment Status 




Control 

Group 


Treatment 

Group 


Total Sample 


Statistical 
Significance 
of Difference 




% 


% 


% 


p-value 


% of children 12 - 23 months 


43.5 


40.9 


42.3 


.74 


% of children 24 - 35 months 


34.2 


34.8 


34.5 


.78 


% of children 36-71 months 


33.6 


35.5 


34.5 


.69 


% homes with any infant 


50.6 


58.0 


54.0 


.47 


% homes with any toddler 


98.8 


97.1 


98.0 


.86 


% homes with any preschooler 


77.8 


82.6 


80.0 


.88 


% homes with majority of children 
< 3 years 


97.5 


97.1 


97.3 


.93 


% homes with majority of children >_3 
years 


2.5 


2.9 


2.7 


.79 


Mix of ages of children enrolled in homes 


One age group only 


7.4 


8.8 


8.0 


.56 


% with only Infants 


0 


0 


0 




% with only toddlers 


7.4 


7.3 


7.3 




% with only preschool 


0 


1.5 


0.7 




Two age groups 


58.0 


44.9 


52.0 


.10 


% with infants/toddlers 


14.8 


10.1 


12.7 




% with toddlers/preschool 


42.0 


33.3 


38.0 




% with infants/preschool 


1.2 


1.5 


1.3 




Three age groups 


34.6 


46.4 


40.0 


.07 


Presence of related children (own/other) 


% homes with provider’s own child 


26.1 


27.6 


26.8 


.82 


% homes with child related to provider 


11.4 


13.2 


12.2 


.90 


% homes with own child and related child 


14.8 


14.5 


14.6 


.93 


Home language background of children 


All children monolingual English 


67.1 


61.8 


64.6 


.79 


All children monolingual Spanish/other 


9.1 


5.3 


6.7 


.14 


Monolingual English & bilingual children 


23.9 


32.9 


28.1 


.06 


Match of provider/children’s language(s)b 


Provider speaks language(s) of all DLLs 


95.2 


76.9 


85.1 


.03 


Accreditation 
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Exhibit 4.8: Characteristics of Child Care Homes in the Analytic Sample after Two Years of 
LeamingGames Intervention by Treatment Status 




Control 

Group 


Treatment 

Group 


Total Sample 


Statistical 
Significance 
of Difference 




% 


% 


% 


p-value 


NA FCC-accredited 


21.4 


18.2 


19.8 


.61 


Sample size 


69 


81 


150 




a Average number of children: control homes = 
(s.d.= 1.85). 


5.45 (s.d.= 1.78); treatment homes = 5.66 (s.d.= 1.94); overall = 5.54 


b In 47 homes with any dual language learners. 











Provider Outcome Measures 

An important analytic task was to develop constructs that measured the provider behaviors that were 
the focus of the LeamingGames program. The observation measures that were administered at 
posttest provided a rich set of data that could be used to assess impacts on providers. However, we 
wanted to develop a smaller number of constructs to avoid problems associated with multiple 
comparisons. When a study examines many outcomes or findings simultaneously, the statistical 
significance of findings may be overstated. Without accounting for these multiple comparisons, the 
likelihood of finding a statistically significant finding increases with the number of comparisons. A 
number of statistical methods can be used to correct for multiple comparisons . 14 The statistical 
methods for correction decrease the likelihood that a finding will be shown to be significant, with the 
need for greater adjustment the more outcomes that are being tested. Even when a correction is not 
applied, keeping the number small reduces concerns about false positives. Therefore, our objective 
was to create a small number of reliable constructs that, based on the items that made up each 
construct, measured outcomes that could reasonably be assumed were the goals of LeamingGames. 

One of the challenges of identifying a small number of key outcomes for LeamingGames lay in the 
scope of caregiver behavior that LeamingGames intends to influence. Our strategy was to develop a 
construct that reflected major components of LeamingGames. Specific constructs included: 

• The amount of time the provider was engaged with individual or pairs of children in 
extended language interactions with cognitively rich content, to assess the provider’s 
engagement in LeamingGames activities (or LeamingGames- like activities); 

• The provider’s availability to children, positive interactions with children, and 
responsiveness, across all activity contexts, to assess the provider’s responsiveness to 
children; and 



14 The traditional approach to correcting for multiple significant tests is the Bonferroni method, which lowers 
the critical p-value for individual comparisons by a factor of 1/m, where m is the total number of 
comparisons. The Benjamini-Hochberg method (Benjamini & Hochberg, 1995) is less conservative than 
the Bonferroni method but is considered by many in the field to protect adequately against Type I error in a 
wide range of applications. 
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• The extent to which the provider supported children’s oral language comprehension, 
across all activity contexts, to assess the provider’s support for learning vocabulary and 
concepts, as in the interactive book reading. 

One of the constructs, provider responsiveness, was based on a single rating scale (the Arnett 
Caregiver Interaction Scale). For the other two outcomes, the process for developing constructs from 
the multiple variables available from the different observation measures involved a multi-stage 
process: 

1 . Identified for each outcome of interest (see above) the variables from the observation 
measures that appeared to address some aspect of that outcome. 

2. Tested the internal consistency of the set of variables from the observation using 
Cronbach’s alpha to determine which variables formed the most psychometrically sound 
construct. 

3. Rescaled all of the variables in the construct on the same metric (i.e., transformed each 
variable into a z-score, with a mean of zero and a standard deviation of one). 

4. Created a score for each provider by summing the z-scores for the variables in the 
construct. 15 

Each stage is described more completely below. 

(1) Identifying observation variables for outcomes. We first reviewed the large set of variables 
from the observation measures to identify a subset that we believed would be related to each of the 
three provider constructs. 

Exhibit 4.9 shows the final set of variables used to build each of the three provider constructs and 
their disposition in the development process. 

(2) Testing internal consistency. For each construct, we tested the extent to which the initial set of 
items in that cluster was in fact measuring the same phenomenon (i.e., the degree of internal 
consistency of the construct). We calculated a Cronbach’s alpha 16 for the construct, and then dropped 
individual items from the construct when doing so increased the value of the alpha. We dropped 
items one at a time, starting with the item that increased our alpha the most, and recalculated our 
Cronbach’s alpha on the remaining set of items. Where the Cronbach’s alpha increased, it meant that 
the reliability of our measure had improved by eliminating a particular variable. In this way, we 
eliminated from the construct any variables whose inclusion reduced the reliability of the construct. 
We repeated this process until we were left with a final set of items where, if any single component or 
item were dropped from the construct, the reliability of the construct measure would be diminished. 

We developed two constructs through this process: (a) rich oral language interaction between 
providers and children and (b) level of support for language comprehension. Exhibit 4.9 identifies the 

15 By definition the sum of the z-scores created in the previous step should also have a mean of zero and a 
standard deviation of one. 

16 Cronbach’s alpha is a statistic often used in social science research to test the consistency of items within a 
domain. It is an indicator of the average correlation of these items. 
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final variables in these first two constructs and the associated Cronbach’s alpha, which ideally should 
be above 0.70. The ratings for the third construct (responsiveness to children) were handled 
differently; the measures for this construct were taken entirely from the Arnett. In the current 
analysis, we used the three constructs that are typically created from the Arnett, based on earlier work 
by the developer. 

(3) Rescaling items and (4) calculating a final scale. For each of the constructs, the individual 
items, which are on different scales, were transformed into a standardized measure (z-score). In 
standardizing each measure, we (1) mean-centered each observation and (2) divided this mean- 
centered value by the standard deviation of the overall measure. Applying this process for each item 
resulted in a uniform measure across items, where each item is expressed as the proportion of 
standard deviations above or below the mean. We then summed these standardized measures into a 
single value which was used as our construct measure. Each provider in the sample was then 
assigned a score for the construct. 

Models for Estimating Impacts 

We used regression models to test the impacts of LeamingGames on provider behavior/activities. 
Specifically, we estimated the regular OLS model shown below to estimate the effect of treatment on 
each provider outcome. 

M 

.+A(7',) + £A,(*-)+'V 

m=2 

where Y k is the outcome measure (e.g. “rich oral language interaction) for the k tb provider, 

T k is an indicator variable, indicating whether the k th provider was randomly assigned to 
LeamingGames, 

X mk is the m th provider characteristic which significantly predicts the outcome (17) for the k tb 
provider, 

r k is a random error term 

Using this model, we estimated the program impact of LeamingGames (i.e. the treatment effect) 
as ff . We ran two models — an unconditional model, without covariates, and the model with a small 
set of covariates — to control for measurable differences between providers and improve the precision 
of our estimated treatment effect. Since including covariates that do not significantly predict the 
measured outcome can actually decrease the precision of our estimated treatment effect, we used 
backward elimination to refine the model, beginning with an initial (complete) set of covariates which 
we believed might predict the selected outcome (l)). 17 In performing this backward elimination, we 
first fit a model using a full set of chosen covariates. Once we estimated this model, we used the 

results to identify the covariate estimate ( fi m ) with the largest p-value (not including the treatment 



17 This issue is a real concern, particularly since our provider-level sample size is very small. 
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estimate, J3 l ) and evaluate its predictive value. If this estimate had a p-value > 0.20, then the 
corresponding covariate was dropped from the model specification, and the new model was refit. 18 

This process was repeated until all covariates with corresponding estimates that did not reach the pre- 
set significance criterion (p < 0.20) were dropped from the model. 

Exhibit 4.10 describes the provider variables used as covariates in the impact models. It should be 
noted that two of the covariates that involved the ages of children in care were measured at the time 
of the posttest observations, which means they were measured after random assignment. It is at least 
theoretically possible that these covariates were affected by the treatment. For example, it is possible 
that LearningGames caused a provider to change the ages of the children that she accepted into care. 
We believe that this is unlikely since LearningGames was designed so that a provider could use it 
with any age. 



18 Our choice of a p < 0.20 cutoff is based on prior research on backward elimination strategies. See 
Maldonado & Greenland (1993), and Budtz-Jorgensen, Keilding, Grandjean.Weihe, & White (2001). 
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Exhibit 4.9: Provider Outcome Constructs and Linked Variables from Observation Measures 


Provider Construct 


Candidate Observation Variables 


Stayed in 
Construct 


Final 

Alpha 


Measure 


Variable(s) 


Rich Oral Language Interaction 


QUEST 


Item 31 ; Extended interaction with child/pairs 


Yes 


.78 


Item 34; Engagement in rich language interactions 


Yes 


Item 39 a ; Extended Rich conversation with child/pairs 


No 


TALK 


Proportion of talk with one child that is extended conversation, focal child only 


Yes 


Proportion of talk with one child that is extended conversation, any child 


Yes 


Snapshot 


Proportion of time where child and provider are in activities which highly involved 
talking 


Yes 


Proportion of time where child and provider are in activities which involve rich 
interaction 


Yes 


Support for Comprehension 


QUEST 


Item 32; Provider nudges child toward new or additional activities 


Yes 


.81 


Item 34; Engagement in rich language interactions 


Yes 


Item 35; Provider encourages activities with books/reading/print 


Yes 


Item 37; Provider makes a variety of material available to child 


Yes 


Snapshot 


Proportion of time children spend in reading (with print) activities 


No 


Proportion of time children spend in early literacy, activities (excluding reading) 


No 


Proportion of time children spend in oral language activities 


No 


Proportion of time children spend in math activities 


No 


RAP 


Number of read alouds for classroom 


No 


Average length of read aloud 


No 


Total time in reading aloud 


No 


Proportion of read alouds with post reading discussion 


No 


Proportion read alouds where provider asks open-ended questions 


No 


Proportion of read alouds with attention to letters/sounds 


No 


Average number of new vocabulary identified in read alouds 


Yes 


Proportion of new vocabulary with comprehension supports (pictures, semantic 
networks) 


Yes 


a This item, by definition, was missing for providers without infants/toddlers, and therefore was dropped from the construct because it was missing for 24 providers with no 
children less than 3 years in their care. 




Exhibit 4.10: Covariates in Provider Impact Analyses 



Variable 


Source 


Definition 


Extent to which provider 
uses high-quality practices 


Baseline provider 
observation: QUEST 


Average rating on 9 subscores (1-3) 


Extent to which provider is 
responsive and engaged 
with children 


Baseline provider 
observation: Arnett CIS 


Average rating on 26 items (1-4) 


Highest educational 
attainment of provider 


Provider background 
questionnaire 


H.S. diploma or less 
Some college 
BA 


Years experience in family 
child care 


Provider background 
questionnaire 


Less than 1 year 
1-3 years 
3-5 years 
More than 5 years 


Child-related specialization 


Provider background 
questionnaire 


No specialization 

Child-related specialization or CDA 


Provider language with 
children during observation 


Baseline provider 
observation: Snapshot of 
Activities 


Spoke > 90 % English 

Spoke > 90 % Spanish 

Spoke a mix of English and Spanish 


One or more infants 
present during observation 


Posttest provider 
observation: Snapshot of 
Activities 


No infant present during observation 
At least one infant present 


Majority of children in home 
ages 3-5 years during 
observation 


Posttest provider 
observation: Snapshot of 
Activities 


Majority of children present during observation are 
> 3 years of age 

Majority of children present during observation are 
3-5 years of age 



Impacts on Providers 

LearningGames had statistically significant impacts on all three provider outcomes. That is, 
compared with the control providers, the LearningGames providers had substantially higher 
frequencies of rich oral language interactions and of interactions presumed to support children’s 
understanding of vocabulary or concepts, and they had significantly higher ratings on their 
responsiveness to the children. The effect sizes for the treatment-control differences were nearly half 
a standard deviation, which by convention is labeled a moderate effect size. Also, none of the 
provider baseline covariates was a statistically significant predictor of the three provider outcomes. 
The consistency of the findings suggests that the LearningGames intervention, despite the apparent 
variability in the extent to which providers implemented a fully realized model, was able to make a 
significant difference in how providers talked to and interacted with the children in their care. 
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Exhibit 4.11 : Impacts of LearningGames on Provider Behavior 





Control 

Group 

Mean 


Treatment 

Group 

Mean 


Treatment 

Effect 


SE 


Statistical 
Significance 
of Impact (t- 
value) 


Rich oral language 
interactions 


-0.18 


0.22 


0.40** 


0.11 


3.72 


Support for development of 
vocabulary/comprehension 


-0.17 


0.20 


0.37** 


0.10 


3.58 


Responsiveness to children 


-0.19 


0.23 


0.47** 


0.16 


2.97 


Key:* = p < 0.05; ** = p<0.01. 
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Chapter 5: Summary of Results and Discussion 



The study of LearningGames was undertaken in the context of policy concerns about the 
development and school readiness of at-risk children who receive their out-of-home care in family 
child care homes. In Massachusetts, children cared for in family child care homes include many of 
our young children who are the most at risk for poor school outcomes (Rulf Fountain & Goodson, 
2008). While many studies have shown that children in family child care homes are generally safe 
and well-cared for, these same studies also document that the learning opportunities for young 
children in family child care typically are not as great as for children in center-based care, especially 
those in public school district programs and Head Start programs (Layzer & Goodson, 2007). Family 
child care is often the care arrangement of choice for families with children under 3 years of age, for 
low-income families, and for families from language and cultural minority groups, especially recent 
immigrants. Because of concerns about the full development and ultimate school readiness of the 
children in our country, especially those who may be at risk for poor academic outcomes, it is 
essential to identify effective strategies to enhance adult-child interactions in family child care that 
help children acquire the important skills that predict long-term school success. Further, recent 
research on language development suggests that the first three years of life may be critically 
important in children’s development, and family child care is and is likely to remain the setting of 
choice for many families with children in that age range even with affordable and culturally 
appropriate center-based options. The LearningGames approach, including its well-documented 
success in the Abecedarian project, appeared well-suited to test in the family child care environment. 

This chapter of the report further discusses the results of the study and their ramifications for policy 
and for future research. The chapter begins with a brief overview of the findings on the impacts of 
LearningGames and the barriers to implementation. The chapter concludes with a discussion of the 
larger issues that the study raises about the challenges and potential of implementing an intensive 
intervention in family child care. Finally, the chapter addresses the value and potential benefits of 
conducting additional investigations of the impact of LearningGames with family child care 
providers. 

Summary of Impacts of LearningGames on Provider Behaviors 

We believe that further study of LearningGames in family care is merited for two major reasons: (1) 
the study was able to show impacts on providers despite substantial variation in implementation of the 
professional development model; and (2) the study was not able to provide evidence of whether or 
not the changes in providers led to meaningful improvements in child outcomes. Both of these points 
are discussed below. 

This study showed that LearningGames had statistically significant positive impacts on the behavior 
of the family child care providers who received up to two years of support in implementing the 
program. LearningGames was effective at promoting high-quality, individualized and small group 
interactions between providers and children, which have been shown in previous research to be 
associated with children’s cognitive and language development. The effect sizes of these outcomes 
ranged from .37 to .47 standard deviations, which the field considers to be of moderate size. 

However, the study could not provide credible evidence about the impacts of LearningGames on 
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children, since the high level of attrition in the child sample and the lack of a baseline assessment to 
verify that observable characteristics across the treatment and control group children assessed posed 
insurmountable threats to the internal validity of the estimates. This was compounded by threats to 
external validity, since there was no way to compare the assessed sample to the original one. 

The logic model for the impact of LeamingGames on children assumes that impacts on providers will 
lead to enhanced outcomes for children. We were unable to test this assumption, but note that a 
number of studies of early childhood interventions have reported impacts on teachers and no 
concomitant impacts on children (for example, Judkins et al., 2008; Gamse, Jacob, JJorst, Boulay, & 
Unlu, 2008; Preschool Curriculum Evaluation Research Consortium, 2008). A recent paper (Garet, 
2008) explored the relationship between teacher behavior and child outcomes in studies that tested 
impacts of interventions, such as LeamingGames, where the causal pathway is assumed to lead from 
teacher professional development to changes in specific provider behaviors with children to improved 
outcomes for those children. Garet’ s research indicates that the strength of this relationship varies as 
a function of the population studied, the ages of the children, the child outcomes, and the method of 
estimating impacts. Garet concludes overall, that an intervention with a large impact on teacher 
behavior (e.g., 1 standard deviation on a quality score) may only have a small impact on students 
(e.g., 0.25 standard deviations). 19 This suggests that if we had been able to examine child impacts for 
LeamingGames, we might have found only small effects, despite the moderate provider-level effects. 
Jlowever, our inability to conduct a valid examination of child impacts is one reason we believe that 
this study does not provide a sufficient test of the effectiveness of the LeamingGames model. 

Implementation of LearningGames\ Barriers and Lessons Learned 

As described in Chapter 3, barriers to implementation limited the intensity of the planned professional 
development model as received by the providers. We believe that this may have affected providers’ 
ability to reach a consistently high level of fidelity to all elements of the LeamingGames model. 
Below we further discuss the barriers to implementation and identify lessons learned. 

The major barriers to implementation fell into three categories: 

• Roles and responsibilities for the LeamingGames implementation. Even initially, the 
lines of authority were complicated and not always clear. They became more complicated 
when the Department of Early Education and Care (EEC) decided that while it would 
endorse the study and provide funding for the intervention, it should not be directly 
involved in the implementation of the LeamingGames professional development model. 



19 Garet estimates that the stable variation in teacher behavior accounts for between 1% and 10% of the total 
variation in student outcomes, which implies a stable between-teacher standard deviation ranging from 0.1 
to 0.32. Extrapolating from these data, Garet argues that an intervention with a large impact on teacher 
behavior of 1 standard deviation on a quality score would only have a small impact on students (e.g., 0.25 
standard deviations). That is, choosing the middle point in the range of between-teacher variance (6%), 
students taught by a teacher who is one standard deviation above average in quality of practices would end 
the year with achievement scores 0.25 standard deviations higher than the scores of students taught by a 
teacher of average quality. 
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• Training LearningGames providers. At least initially, home visitors met with resistance 
on the part of some of the family child care providers, some of whom believed that they 
were already using the approach because the techniques were deceptively simple. Others 
believed that it was not possible to work with one or two children. However, at the end of 
the study, home visitors reported that providers in the LearningGames homes had by and 
large mastered the individualized approach. In addition, there was some resistance among 
some providers to being subjected to bimonthly hour-long TA visits as well as some 
aspects of the protocol. Providers also reported that they desired getting direct training 
from the developer and many attended direct training sessions held in Y ear 2 of the study. 

• Train-the-trainer approach using home visiting staff. The study did not anticipate the 
amount of time that would actually be needed for home visitors to master the 
LearningGames approach. Even more time appeared to be needed for home visiting staff 
with relatively low levels of educational attainment or knowledge of child development. 
Providing home visitors with adequate time for mastery of the approach was further 
impeded by high home visitor caseloads and a high level of turnover among home 
visitors. 

• Support by project coordinators and MindNurture staff. Since project coordinators 
received training at the same time as did the home visiting staff, they did not have the 
opportunity to master the approach before they were asked to support its use by the home 
visiting staff. In addition, in the first year of the study, while Dr. Sparling and his staff 
provided more training than was originally planned, MindNurture did not receive the 
level of resources necessary for staff to be in the state providing direct support to the 
project coordinators and networks to the degree needed, in large part because these needs 
were not well understood prior to implementation. More funding added in Y ear 2 
facilitated his ability to provide more technical assistance to staff as well as direct 
training to providers. 

Although these issues were serious, and four networks withdrew from the study, it is important to 
point out that most of the network staff believed that the study was worthwhile, were glad to 
participate in it, and did what they could to address the issues described above. Network leadership 
supported the LearningGames approach and several reported planning to continue to use it with 
family child care providers after the study had ended. 

Indeed, despite barriers, many providers received periodic home visits that supported their practice of 
LearningGames. Network staff and others reported that at least some of the caregivers had mastered 
the LearningGames approach. When asked to complete fidelity ratings, they scored most of their 
homes as implementing most aspects of LearningGames. Finally, a separate fidelity rating developed 
by Abt staff indicated that the LearningGames providers were more likely to exhibit behaviors that 
are closely tied to successful implementation. 

Implementation Lessons Learned 

The fact that LearningGames resulted in positive provider impacts leads us to recommend further 
exploration of the suitability of LearningGames in family child care homes. If LearningGames were 
implemented again in this setting, we have a number of recommendations on how it should be 
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undertaken to address systemic issues about the implementation of strategies that use a train-the- 
trainer approach, about the LeamingGam.es model itself, and about implementing high-intensity 
interventions in family child care homes. 

Addressing Barriers to Implementing the Professional Development Model 

Once LearningGames was selected as the intervention to be tested in Massachusetts, Abt worked with 
the developer to determine the best approach to professional development, given the level of 
resources available for training, the distance between the developer’s home base and the provider 
sites, and the organization of the family child care networks within which the intervention would be 
implemented. The train-the-trainer model and the focus on in-home coaching of the providers by the 
home visitors were selected because they seemed like the best fit within the budgetary and 
organization constraints of the study. Since this was the first major “real world” implementation of 
LearningGames in family child care, there were many implementation issues that were not fully 
anticipated but could be addressed in future efforts. Some suggestions include the following: 

• Build in more time for training the trainers. More time is needed than the study 
provided for the initial training of home visiting staff on LearningGames and for the staff 
providing support to home visitors. 

• Lower home visitor caseloads. If home visitors are expected to take on the work of 
training providers on a specific curriculum such as LearningGames, the home visitor 
caseloads should be lowered sufficiently to allow home visitors to have the needed time 
to do the standard LearningGames protocols, although not be so low as to make real-life 
implementation of the model infeasible. Lower caseloads are needed to ensure that there 
is sufficient time for the home visiting staff to prepare for, conduct, and document the 
visit. In addition, home visitors from networks have additional technical assistance 
duties that are separate from the LearningGames protocol, which means that a visit needs 
to be longer than one hour. 

• Address home visitor turnover. Factors in home visitor turnover that are within the 
control of the networks should be addressed. For example, it would be possible that 
networks would ensure that home visitors would not be moved from their roles during the 
intervention except under special circumstances. In addition, in other studies, 
professional development staff have been provided with a bonus for staying in their role 
for at least one year; this strategy could be used here. 

• Develop a system for training new home visitors. A formal system of training new home 
visitors needs to be in place so that new visitors could be retrained sufficiently so that 
their providers would not lose ground during the rehiring and retraining period. 

• Augment the train-the-trainer model with direct training. The train-the-trainer 
approach should be augmented, especially in the first year, with direct training of 
providers if at all possible. Direct training could serve to jump-start providers and build 
their enthusiasm for the endeavor. While direct training is likely to be a very helpful 
augmentation, ongoing in-home instruction still seems fundamental. 

• Focus more on fidelity measurement. Fidelity of the professional development model 
should be formally measured and information should be used to document issues to make 
midcourse corrections, as needed. 



Massachusetts Family Child Care Study: Evaluation of LearningGames 



71 





Some of the issues above could be addressed with additional resources for the LearningGames 
implementation. While family child care networks received financial support from the Department of 
Early Education and Care, they did not receive any direct funding specifically for LearningGames. 
Networks still attempted to use their ongoing funding for these additional responsibilities; however; 
we believe that future implementations of LearningGames could be enhanced if networks and other 
technical assistance providers received financial support that was sufficient to pay for the additional 
time needed for home visiting staff and others. 

Addressing Barriers to Provider Acceptance of the LearningGames Model 

LearningGames aims to change fundamentally adults’ interactions with children, weaving rich oral 
language and developmental scaffolding throughout all daily activities. It also seeks to strengthen a 
unique and positive aspect of many family child care settings: a smaller adult/child ratio that can be 
capitalized upon for the individualized LearningGames approach. However, there was initial 
resistance to the approach by many family child care providers and by some home visitors. First, the 
games and instructions were deceptively simple and many providers perceived that they were “doing 
these things already.” In addition, network staff and home visitors reported that providers initially did 
not believe that individualized approaches were feasible given the many pressing demands on 
providers, such as feeding and toileting very young children. For some providers, this initial 
resistance to playing the simple games changed as they saw how children reacted to LearningGames 
approaches. In addition, home visitors reported that the resistance to individual-level interaction 
changed as providers saw how it was possible to take time to focus on a specific child. It might be 
possible for the developer to include explicit training on strategies for organizing children into 
multiple activities and/or adding components on building children’s self-regulation skills so that the 
provider can work with only some children and feel comfortable that other children are engaged and 
safe. 

One of the great appeals of LearningGames is that it includes activities and strategies for children 
from birth through 5 years, since this, in theory, addresses one of the challenges for family child care 
providers — the expectation that they can provide appropriate stimulation for all age groups, as 
opposed to center-based teachers who work with children at similar developmental levels. There are 
relatively few curricula available for children under 3 years of age, which is one of the reasons that 
LearningGames is attractive for family child care homes, which often serve younger children. In fact, 
the strongest results for LearningGames, from the Abecedarian study, were for children under 3 
years. For older preschool children in family child care, there may be alternatives to LearningGames 
suitable for family child care in the form of evidence-based curricula, which might offer more 
comprehensive support for children’s development. For example, curricula such as Breakthrough to 
Fiteracy 20 or Tools of the Mind 21 work on a range of skills (e.g., early math, early literacy, self- 
regulation). Some of these curricula, in whole or in part, might be adaptable for use in family child 
care homes, so that the children in home-based care could receive the benefit of more intensive 
instructional support. We believe that additional research should compare the LearningGames 



20 Published by Wright Group/McGraw Hill, http://www.breakthroughtoliteracv.com/ 

21 Published by Metropolitan State College of Denver. 
http://www.mscd.edu/extendedcampus/toolsofthemind/curriculum/index.shtml 
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approach for preschool children with such other curricula and evaluate the possibility of using other 
approaches instead of or alongside LeamingGames. 

Future efforts could also address some provider criticisms of some of the aspects of the home visiting 
model. Anecdotal evidence indicates that some providers reported being uncomfortable and 
sometimes judged when they were watched by home visitors and other staff as they interacted with 
children. They also described being uncomfortable with some specific components of the protocol. 
These issues could be addressed by training home visitors to help them with specific techniques as 
well as testing different LeamingGames home visit protocols and getting feedback from providers 
about what was most helpful to them. 

Addressing Turnover Among Providers and Children 

Even if the train-the-trainer approach is sufficiently funded, staffed, and planned, and even if the 
LeamingGames approach were modified to facilitate its earlier adoption by providers, the issues of 
turnover among family child care providers and children in care would hinder the effectiveness of this 
or any other intervention in family child care settings. If the field is to pursue quality enhancements 
in family child care, these issues need to be addressed. For the Massachusetts study, we attempted to 
include those providers who were most likely to remain in the field the longest. There were providers 
who dropped out because they no longer wanted to be in the study. Flowever, there also were many 
who stopped being family child care providers, had no children in care for significant periods of time 
or left the family child care network and therefore no longer received technical assistance visits. 

While turnover is a problem among child care center teachers, the center remains in operation when a 
teacher leaves and therefore there is generally still a way to reach the same children; this is not the 
case in family child care. 

A second issue to be addressed relates to turnover among children. Unlike center-based arrangements, 
where children frequently “graduate” from one classroom to the next and must form new 
relationships with caregivers each year even when there is no teacher turnover, family child care 
offers the opportunity for ongoing and sustained interactions between adults and young children, 
particularly those who enter care as infants and toddlers. Our study indicates that despite this 
potential, nearly all of the children who were in care at baseline were not there after two years. Some 
of this turnover is to be expected and appropriate, as children reach school age or move to center- 
based care; however, even for the children who were less than 3 years of age at baseline, who could 
still be with the provider two years later, more than 90% were not in the same homes two years later. 
This suggests that many children in a family child care home would not have the opportunity to 
receive the benefit of a sustained intervention. Instability for children also potentially has 
consequences for providers, who are challenged to establish a consistent program to support 
individual children’s learning and development. More needs to be learned about continuity of 
children’s experiences in family child care and whether policies can be changed to enhance 
continuity. 

The Future for LeamingGames in Family Child Care 

In summary, we suggest that there is a rationale for further study of LeamingGames in family child 
care, if certain conditions can be met in support of full implementation of the intervention and for 
ensuring the integrity of an evaluation study. In terms of implementation, LeamingGames should 
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have a more resource-intensive support system and some redesigned training strategies to address 
barriers to the implementation of the model. In terms of the evaluation design, a data collection 
approach for measuring child outcomes needs to take into account the high level of turnover among 
children. A continuous system of child assessment to gather baseline information when children enter 
care, measure children’s outcomes before they leave a provider, and follow up with children after 
they leave a provider is most desirable. However, in spite of these improvements there is concern that 
these changes to the implementation of LearningGames and to future evaluation designs may not be 
cost effective if fundamental issues of family child care turnover are not addressed. In the current 
study, in spite of recruitment of providers who were associated with networks and who had been in 
business for at least two years, (a relatively stable group of providers) the study suffered from 
significant levels of attrition. 
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Exhibit A.1 Sample at Baseline and Attrition Over Time by Agency 





Sample of Homes Recruited 


Sample of Homes 
Participating 


Sample of Homes 
Participating 


Sample of Homes 
Participating 


% Attrition from Recruited 
Sample 


(spring 2005) 


(Fall 2006 Observations) 


(Fall 2007 Observations) 


(Winter 2008 Assessments) 


(Observations Fall 2007) 


T 


C 


Total 


T 


C 


Total 


T 


C 


Total 


T 


C 


Total 


T 


C 


Total 


Agency 1 


5 


5 


10 


4 


5 


9 


4 


4 


8 


3 


4 


7 


0.2 


0.2 


0.2 


Agency 2 


5 


6 


11 


3 


2 


5 


2 


2 


4 


1 


2 


3 


0.6 


0.67 


0.64 


Agency 3 


10 


10 


20 


9 


7 


16 


8 


7 


15 


7 


6 


13 


0.2 


0.3 


0.25 


Agency 4 


5 


6 


11 


5 


6 


11 


5 


5 


10 


5 


3 


8 


0 


0.17 


0.09 


Agency 5 


6 


6 


12 


5 


5 


10 


4 


3 


7 


4 


2 


6 


0.33 


0.5 


0.42 


Agency 6 


11 


11 


22 


9 


8 


17 


7 


9 


16 


5 


5 


10 


0.36 


0.18 


0.27 


Agency 7 


11 


9 


20 


8 


9 


17 


6 


8 


14 


5 


7 


12 


0.45 


0.11 


0.3 


Agency 10 


9 


11 


20 


6 


6 


12 


0 


0 


0 


0 


0 


0 


1 


1 


1 


Agency 1 1 


9* 


5 


14 


6 


2 


8 


4 


1 


5 


4 


0 


4 


0.56 


0.8 


0.64 


Agency 12 


17 


22 


39 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


1 


1 


Agency 13 


7 


7 


14 


5 


6 


11 


3 


6 


9 


3 


3 


6 


0.57 


0.14 


0.36 


Agency 14 


16 


17 


33 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


1 


1 


Agency 15 


13 


14 


27 


5 


10 


15 


3 


9 


12 


3 


8 


11 


0.77 


0.36 


0.56 


Agency 16 


3 


4 


7 


1 


4 


5 


0 


1 


1 


0 


1 


1 


1 


0.75 


0.86 


Agency 1 7 


7 


7 


14 


3 


6 


9 


2 


6 


8 


2 


6 


8 


0.71 


0.14 


0.43 


Agency 18 


11 


11 


22 


10 


8 


18 


10 


8 


18 


9 


8 


17 


0.09 


0.27 


0.18 


Agency 19 


5 


5 


10 


3 


5 


8 


2 


4 


6 


2 


4 


6 


0.6 


0.2 


0.4 


Agency 20 


6 


7 


13 


4 


4 


8 


2 


3 


5 


1 


2 


3 


0.57 


0.67 


0.62 


Agency 21 


9 


9 


18 


6 


5 


11 


6 


5 


11 


5 


1 


6 


0.33 


0.44 


0.39 


Agency 22 


8 


8 


16 


7 


7 


14 


7 


7 


14a 


0 


0 


0 


1 


1 


1 


All Agencies 


173 


180 


353 


99 


105 


204 


76 


88 


164 


59 


62 


121 


0.43 


0.38 


0.41 



d This agency dropped out of the study after the provider observations but before the child assessments. For the assessments, therefore, the sample is smaller by 14 homes, which 
raises the attrition rate to 46% overall, 45% for LearningGames providers and 40% for control providers. 



* Five of these providers were originally randomly assigned to LearningGames within agency 12 but switched to agency 1 1 and continued study participation before implementation 
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Appendix B 



Child Impacts 

As stated earlier, we concluded that because of the high sample attrition and the lack of baseline data 
to adjust for potential bias in the analytic sample, the study could not provide credible estimates of 
child impacts. As exploratory analyses, we did examine child impacts at the end of Year 2, after 
providers had the opportunity to implement LeamingGames for two years. 

Child Sample 

To participate in the child assessments, a child had to meet two eligibility criteria: 

1 . The child had to have been with the family child care provider for at least six months 
prior to the assessments (since approximately July 2007). 

2. The child had to have signed parent consent to be part of the assessments. 

The six-month criterion was implemented to insure that children in the assessments had been exposed 
to their provider for a sufficient time to expect any effects of LeamingGames. It should be noted that 
this criterion also meant that children assessed were all at least six months of age. 

The data collection plan included that four children were to be assessed from each family child care 
home, if possible: two children under 3 years of age and two children between 3 and 5 years of age. 

If there were not enough eligible children in one of the two age groups, more children could be 
selected in the other age group, up to four children total. If there were fewer than four eligible 
children in the home, all eligible children were tested regardless of age. 

Assessments were conducted on 374 children, representing 121 (34%) of the originally-assigned 353 
providers. As described in Chapter 2, 19 (13%) of the 150 providers who remained in the study at 
posttest did not have any children who were assessed, either because there were no children of 
eligible age in the home or because the provider did not collect any signed parent permissions. In the 
majority of the homes in the sample, at least one child in each of the two age groups was assessed 
(Exhibit 4.9). 



Exhibit B-1 Providers in the Child Assessment Sample by Treatment Status 




Control 

Group 


Treatment 

Group 


Total 

Sample 


Statistical 
Significance 
of Difference 
(p- value) 


Only children < 3 years 


16 % 


17 % 


16 % 


.87 


Only children 3-5 years 


21 % 


12 % 


16 % 


.11 


At least 1 child in each age 
group 


63 % 


70 % 


66 % 


. 24 . 












Sample 


62 


59 


121 
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The child sample included more females than males, in both the treatment and the control group 
(Exhibit B-2). Most children were at least 30 months old. The majority of children in the sample 
came from homes where English was spoken, but more than a third of the children came from homes 
where another language was spoken (usually Spanish). Most of these children were bilingual and 
spoke at least some English in the family child care homes. A quarter of the children in the sample 
spoke only Spanish. The sample included only a small number of children who were related to the 
provider. Overall, 8 percent of the children were related to the provider — about half of these were the 
providers’ own children and the other half were nieces, nephews or grandchildren. 



Exhibit B-2 Characteristics of Child Sample at Posttest 




Control 

Group 


Treatment Group 


Total 

Sample 


Statistical 
Significance of 
Difference 


% 


% 


% 


p-value 


Sex 










Male 


45 


46 


45 


.95 


Age 








.39 b 


6-12 months 


3 


6 


5 




13-24 months 


20 


20 


19 




25-36 months 


34 


31 


32 




37-48 months 


25 


24 


24 




49-60 months 


21 


15 


18 




60-72 months a 


1 


3 


2 




Home language 








,59b 


Monolingual English 


65 


61 


63 




Bilingual 


12 


13 


12 




Monolingual Spanish 


23 


27 


25 




Related to provider 










Provider’s own child 


5 


5 


4 


.88 


Other family relationship 


5 


5 


5 


.91 












Sample 


62 


59 


121 




a Children not yet in kindergarten 
b Chi-square test 



Child Outcome Measures 



The child outcomes were standard scores calculated from the two standardized assessment measures 
administered in the study: the Preschool Language Scale, Fourth Edition: Auditory Comprehension 
(PLS-4: Zimmerman, Steiner & Pond, 2002) and the Bracken Basic Concept Scale-Revised (BBCS-R 
SRC: Bracken, 1998). The PLS-4 was administered to all children in the sample starting at 6 months 
of age. The BBCS-R also was administered to children who were at least 30 months of age. 

On the PLS-4, the outcome measure was the standardized score for Auditory Comprehension (mean = 
100, standard deviation = 15), which is age-adjusted. On the BBCS-R, the outcome measure was the 
standardized score for the School Readiness Construct. 
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Estimated Impacts on Children 

Models for Estimating Impacts 

Hierarchical linear regression models (HLM) were used to test for impacts on child outcomes, to 
account for the clustering of children within family child care home. 1 We estimated three HLM 
models. Model 1 (the unconditional model) estimated the treatment effect without any additional 
child- or provider-level co variates. Model 2 estimated the treatment effect and includes the child- and 
provider-level covariates. Model 3 included the same covariates and treatment interactions with the 
covariates. The models included four covariates at the child level (sex, age in months at time of 
testing, relationship to the provider, and home language) and four covariates at the provider level 
(provider education, the majority age of children in the home, if there was at least one infant in care 
and if there was at least one pre-school age child in care). Exhibit B-3 describes the values taken by 
the covariates tested in the child impact models. Since the standardized scores on the two child 
assessments represented age-adjusted scores, there was no need to include age separately as a 
covariate. 

The equations below specify the HLM model (model 2) for testing impacts where children (level 1) 
arc nested within providers (level 2). 

Level- 1 Model: 

M 

r, =Au.+ZA„<V,,,) + '> 

m = 1 

Level-2 Model: 

N 

Pok = Too + f 01 (Tic ) + ^ y On (Z„k ) + U 0k 

n = 2 

Pmk = y mO + Yml( T k) 

r jk ~ N (0,cr 2 ) 

u 0k ~ N( 0,T 0k ) 

where Y jk is the outcome measure (i.e. either the PLS-4 or Bracken score) for the/ h 
child nested within the k th provider, 

T k is an indicator variable, indicating whether the k th provider was randomly assigned to 
receive the LG curriculum, 

X m j k is the / 77 th child-level characteristic which may predict overall test score for 
the / h child nested within the k th provider, 

Z nk is the (n - 1) provider-level characteristic which may predict differences in 
average outcomes 



Hierarchical linear modeling helps correct for potential complexities that arise from the clustered design 
(i.e. intercluster correlations, heteroscedasticity, etc.) and adjusts the standard errors of our estimates 
normally obtained through an ordinary OLS model. 
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In this model, we are most interested in the average estimated treatment effect, r , which is expressed 

M _ 

as y m + I y ml X . This estimate tells us the expected impact of Learning Games on language 

m= 1 

assessments for the average child who received the intervention. The average child ( X ) in this case 
is represented by the average characteristics of the sample of children included in the estimation of Y. 
In other words, where the dependent variable (T) is a child’s standardized score on the Bracken SRC, 
then the estimated t represents the impact of treatment on the average Bracken-tested child, who is 
different from the average PLS4-tested child. 

Baseline measures of child and provider-level covariates were included in the models to control for 
measurable differences between children and to improve the precision of our estimated treatment 
effect. In the case of the child outcomes, we were also interested in determining whether the 
observed characteristics of children significantly predicted differences in outcomes consistent with 
our theory about their potential explanatory power in predicting differences. Therefore, the HLM 
models included all of the covariates in order to evaluate the estimates for all of the covariates even 
where they were not significant predictors of our child outcomes. 



Exhibit B-3 Covariates in Child Impact Analyses 


Variable 


Level 


Description 


Sex 


Child-level 


Female 

Male 


Language 


Child-level 


English is primary language 
Spanish is primary language 
English-Spanish bilingual 


Age 


Child-level 


Younger than 30 months 
30 months or older 


Education level of provider 


Provider-level 


H.S. diploma or less 
Some college 
BA or more 


One or more infants present in home 


Provider-level 


- Infant is present 

- No Infant present 


Majority age of children in home 


Provider-level 


- Majority is pre-K 

- Majority is not pre-K 



Results 

Exhibit B-4 summarizes the findings from the regression models estimating the impact of 
LeamingGames on child outcomes, based on the unconditional model without covariates. On the 
PLS-4, the difference in performance for the treatment and control children is not statistically 
significant. On the BBCS-R School Readiness Construct, the preschool children in LeamingGames 
homes scored, on average, significantly lower relative to children in the control group homes (p < 
.01). As a standardized effect, the estimated treatment effect size on the BBCS-R SRC is 0.43 
standard deviations, which is generally considered to be a moderate effect. 

When the covariates were added to the models, the direction and magnitude of the effect sizes were 
not affected (Exhibit B-5). Exhibit B-6 shows detailed results from HLM models including 



Abt Associates Inc. 



Evaluation of LeamingGames -Appendices 



6 





covariates. The covariates themselves were related to child outcomes in expected ways. Boys scored 
significantly lower than girls on both the PLS-4 and the Bracken. Similarly, children who were 
bilingual or monolingual Spanish-speaking scored significantly lower on the PLS-4 and the Bracken 
than children who spoke only English (Exhibit B-7). 



Exhibit B-4 Estimated impacts of LearningGames on Child Outcomes: Unadjusted HLM 
Model 




Standardized 
Effect Size 


Treatment 

Effect 


SE 


t-stat 


PLS-4 Auditory Comprehension Subscaie 
(full child sample) 


0.23 


-3.65 


1.87 


1.95 


BBCS-R School Readiness Composite 
(children over 30 months) 


0.43 


-7.82“ 


2.80 


2.79 


Note: * = p<0.05; ** = p<0.01 





Exhibit B-5: Estimated Impacts of LearningGames on Child Outcomes based on Three HLM 
Models 




Standardized 
Effect Size 


Estimate (t) 


SE 


t-stat 


Covariates in Model 


Child- and 
Provider-Level 


Treatment 

Interactions 


PLS-4 Auditory Comprehension 3 


Model 1 Treatment Effect 


0.23* 


-3.69 


1.93 


-1.91 






Model 2 Treatment Effect 


0.20* 


-3.24 


1.76 


-1.84 


X 




Model 3 Treatment Effect 


0.21* 


-3.26 


1.76 


-1.85 


X 


X 


Bracken School Readiness Construct 


Model 1 Treatment Effect 


0.44*** 


-7.99“ 


2.76 


-2.89 






Model 2 Treatment Effect 


0.42*** 


-7.75“ 


2.70 


-2.87 


X 




Model 3 Treatment Effect 


0.42*** 


-7.72“ 


2.72 


-2.84 


X 


X 


“ Two cases were dropped from analysis 
One case was dropped from analysis 
* = p < < 0.05; ** = p < 0.01 
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Exhibit B-6 Estimated Effects of Covariates in Child Impact Models 





PLS-4 


Bracken 


Model 


(2) 


(3) 


(2) 


(3) 


Covariates 










Intercept 


110.09** 


107.65** 


94.38** 


91.66** 












Child Characteristics 










Male 


-4.87** 


-4.41* 


-3.45 


-4.69 


Age in months 


-0.06 


-0.03 


0.21* 


0.29 


English speaking home 


-8.02** 


-4.82 


-8.85** 


-10.35* 


Bilingual home 


-5.80* 


-6.02 


-8.97** 


-9.12 












Provider Characteristics 










PreK-age child present in home 


-7.83** 


-7.59** 


- 


- 


Infant present in home 


— 


— 


0.39 


-0.45 












Treatment and Treatment Interactions 










Treatment 


- 


0.56 


-7.75** 


-2.34 


Treatment *Male 


- 


-0.74 


- 


2.44 


Treatment Age 


- 


-0.06 


- 


-0.16 


Treatment * Spanish speaking home 


— 


-5.67 


- 


3.17 


Treatment *Bilingual home 


- 


0.27 


- 


0.68 












Sample 


366 


219 


* = p < < 0.05; ** = p < 0.01 



Exhibit B-7 Average Differences on Child Outcomes for Subgroups of Children 


Group 


Average Score 


PLS-4 

Auditory Comprehension Subtest 


BBCS-R 

School Readiness Composite 


Male 


-6.7** 


-3.2 


Spanish-speaking home 


-7.3“ 


-8.6“ 


Bilingual home 


-5.7* 


-8.2 








Sample 


366 


219 


* = p < 0.05) (** = p < 0.01 



Results by Age of Child 

We analyzed the scores on the PLS-4 by age of child, to investigate whether the impacts of 
LeamingGames varied by age of child. We then added a covariate to the models to test the effect of 
mixed age groups on children’s performance. For the test of impacts for younger children on the 
PLS-4, we added a covariate indicating whether or not there was a preschool-age child in the home as 
well as the younger child. For the tests of impacts for the older children on the PLS-4 and the 
Bracken, we added a covariate indicating whether or not there was an infant or toddler in the home as 
well as the preschooler. The results are shown in Exhibit B-8. Exhibit B-9 provides detailed results 
from FILM models for subgroups of children.) 
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Preschool children. On the PLS-4, for the preschool children, there was no impact of 
LearningGames. As was true for the full sample, males scored less well compared with females, and 
children from homes where another language than English was spoken performed less well compared 
with children from monolingual English-speaking homes. 

Infants/Toddlers. On the PLS-4, for the infants and toddlers, there was no impact of 
LearningGames. Boys scored less well compared with girls, and there was also a negative effect of 
being from a monolingual Spanish-speaking home. The presence of a preschool-age child also was 
negatively related to scores on the PLS-4. 



Exhibit B-8 Estimated Impacts of LearningGames on the PLS-4 by Age of Child 




Infants & toddlers 


Preschoolers 


Treatment 


-2.64 


-3.80* 








Child Characteristics 






Male 


-7.97** 


-3.39 


Age in months 


-0.25 


-0.19 


Monolingual Spanish home 


-7.71* 


-8.55** 


Bilingual home 


-1.70 


-9.46** 








Provider Characteristics 






PreK-age child present in home 


-10.50** 


-- 


Infant present in the home 


- 


0.39 








Sample 


149 


219 


* p < 0.05; ** p < 0.01 
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Exhibit B-9 Estimated Effects of Covariates on Child Impacts for Subsamples of Children by Age 





PLS-4 


PLS-4 


PLS-4 


Bracken 


Full Sample 


Children < 3 Years 
(PLS-4 Only) 


Children >3 Years 
(PLS-4 and Bracken) 


Full Sample 


Model 2 


Model 3 


Model 2 


Model 3 


Model 2 


Model 3 


Model 2 


Model 3 


Covariates 


















Intercept 


110.09** 


107.65** 


116.89** 


117.50** 


109.42** 


105.51** 


94.38** 


91.66** 




















Child Characteristics 


















Male 


-4.87** 


-4.41* 


-7.97** 


-5.90 


-3.39 


-3.26 


-3.45 


-4.69 


Age in months 


-0.06 


-0.03 


-0.25 


-0.38 


-0.193 


-0.11 


0.21 


0.29 


English speaking home 


-8.02** 


-4.82 


-7.71* 


-5.28 


-8.55** 


-6.66 


-8.85** 


-10.35* 


Bilingual home 


-5.80* 


-6.02 


-1.70 


-0.05 


-9.46** 


-11.10* 


-8.97** 


-9.12 




















Provider Characteristics 


















PreK-age child present in home 


-7.83*** 


-7.59*** 


-10.50** 


-9.86** 


— 


— 






Infant present in home 


— 


— 


— 


— 


0.39 


1.26 


-0.45 


2.69 




















Treatment and Treatment Interactions 


















Treatment 


-3.24* 


0.56 


-2.68 


-5.36 


-3.80* 


3.53 


-7.75*** 


-2.34 


Treatment *Male 


— 


-0.74 


— 


-3.52 


— 


-0.35 


— 


2.44 


Treatment *Age 


— 


-0.06 


— 


0.25 


— 


-0.16 


— 


-0.16 


Treatment *Not English speaker 


— 


-5.67 


— 


-3.69 


— 


-3.17 


— 


3.17 


Treatment *Bilingual speaker 


— 


0.27 


— 


-2.61 


— 


3.06 


— 


0.68 




















Number of Observations 


366 


149 


217 


219 


* = p < < 0.05; ** = p < 0.01 



O 




Sensitivity Analyses 

Outliers. Because we were concerned with undue effect of children with very low or very high 
scores on the assessments, we tested the sensitivity of our estimated treatment to the influence of 
outlying observations by re-estimating the models without observations where test scores are at the 
extreme values. When two observations were dropped from the analysis of the PLS-4 and one 
observation from the Bracken, the estimates of the treatment effect were not significantly different 
from our estimates using the full sample, after omitting these extreme values. Exhibit B-10 shows the 
results of the HLM models excluding outliers. 

Score Discrepancies. Another cause for concern in the data is when a child’s scores on the PLS-4 
and the Bracken are very different. Widely varying scores may be indications or evidence of 
unreliable measures of child outcomes. Therefore, we also tested the sensitivity of our estimated 
treatment to the exclusion of observations where a child’s scores on the two tests varied widely. This 
was defined as an absolute difference between PLS-4 and Bracken scores greater than 15 points 
(approximately one standard deviation of the nationally normed test). It should be noted that this only 
applies to the older children who were administered both tests. The impacts models were re- 
estimated excluding observations with large differences, which involved dropping 56 observations 
from the analysis of the PLS-4 and 53 observations from the analysis of the Bracken. The new 
estimates were not significantly different from our estimates using the full sample. Exhibit B-ll 
provides detailed results of the HLM models excluding cases with discrepant scores. 



Exhibit CIO Estimated Treatment Effect of LearningGames on Child Outcomes, with Outliers 
Excluded 




Standardized 
Effect Size 


Estimate (t) 


SE 


t-stat 


Covariates Used 


Child- and 
Provider Level 


Treatment 

Interactions 


PLS-4 Auditory Comprehension a 














Model 1 Treatment Effect 


0.24 


-3.65 


1.94 


-1.88 






Model 2 Treatment Effect 


0.21 


-3.24 


1.77 


-1.83 


X 




Model 3 Treatment Effect 


0.21 


-3.27 


1.77 


-1.85 


X 


X 


Bracken SRC a 














Model 1 Treatment Effect 


0.42** 


-7.64** 


2.70 


-2.83 






Model 2 Treatment Effect 


0.41** 


-7.42** 


2.65 


-2.80 


X 




Model 3 Treatment Effect 


0.41** 


-7.37** 


2.67 


-2.76 


X 


X 


“ Two cases were dropped from analysis 
b One case was dropped from analysis 

* = p < < 0.05; ** = p < 0.01 
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Exhibit B-11 : Estimated Treatment Effect of LearningGames on Child Outcomes, Excluding 
Cases with Discrepant Scores on Two Outcomes 3 




Standardized 
Effect Size 


Estimate (t) 


SE 


t-stat 


Covariates Used 


Child-and 
Provider Level 


Treatment 

Interactions 


PLS-4 Auditory Comprehension b 


Model 1 Treatment Effect 


0.20 


-3.20 


2.16 


-1.48 






Model 2 Treatment Effect 


0.17 


-2.71 


1.93 


-1.41 


X 




Model 3 Treatment Effect 


0.17 


-2.76 


1.94 


-1.42 


X 


X 


Bracken School Readiness Construct 0 


Model 1 Treatment Effect 


0.36* 


-5.88** 


2.99 


-1.97 






Model 2 Treatment Effect 


0.31* 


-5.09* 


2.89 


-1.76 


X 




Model 3 Treatment Effect 


0.31* 


-5.00* 


2.92 


-1.71 


X 


X 


a Defined as scores that are more than 1 standard deviation (15 points) different. 
b 56 cases were dropped from analysis 
c 53 cases were dropped from analysis 

* = p < < 0.05; ** = p < 0.01 
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Appendix C: Coding Forms 
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Snapshot of Activities: Family Child Care Home - Fall 2007 





Snapshot #31 

Time: : am pm 



Key — Provider/Asst. Language: 

E = English [circled = rich] 
S = Spanish 
O = Other 

Provider Involvement: 

S - active; L = letters/ #s used 



Describe Activity 



1 Reading /text /vocabulary (w /print) 

2 Early literacy (non-reading) 

3 Oral language & basic concepts 

4 Early math 

5 Developmental activities 



6 Group/meeting time 

7 TV/video /computer 

8 Gross motor play 

9 Physical care/self-help/ comforting 

10 Meals /snacks 

11 Transition / routines/ management 

12 Not involved in other listed activity 

13 In restraints 

14 Sleeping 



Other: 








FCC-SNAPSHOT Definitions 




1. Reading /text /vocabulary (with print) 

• Looking at books or pictures, adult reading aloud, children 
reading together without adult, emergent reading (pretending to 
read), shared reading activities. 

• Listening to stories on audiotape or CD. 

• Teaching children new vocabulary words with print support for 
vocabulary (e.g., printed word). 



2. Early literacy activities (non-reading) 

• Alphabet/letter knowledge — recognizing letter/numeral forms, 
letter /sound correspondence 

• Sounds/sinqinq/rhymes/poems 

• Emergent writing /copying /tracing — Child writing includes 
pretend writing, scribbling, invented spelling. Child dictation to 
teacher; tracing letter/number templates. Practice in correctly 
writing numerals/distinguishing numerals. 



3. Oral language & basic concepts instruction/game 

• Discussions, new concepts, and vocabulary. 

• Provider with 1 or 2 children, in structured cognitive game 
(labeling, naming, identifying, discussing). 



4. Early math activities 

• Math concepts /attributes /colors — formal and informal 

communication of math concepts, attributes, or colors. Shapes, 
counting, measuring, patterns, amount. Identifying and matching 
non-geometric shapes (animals, familiar objects). Identifying 
and matching colors and color names. 



5. Developmental activities 

• Science /nature — formal and informal communication of science 
or nature. Examples: astronomy, working with pets, collecting 
leaves, feeding pets, magnets, health & safety. 

• Dramatic play — p retend or make-believe play; dress-up, playing 
with dolls; assigning roles; zooming cars and trucks. Includes 
acting out stories /playing with puppets, figures of people/ 
animals, and stuffed animals in pretend worlds. 

• Creative play / Arts and crafts — creating visual art (painting, 
drawing, sculpting clay & play dough, cutting and pasting). 
PlayDoh® 

• Music — usingjnstruments, formal and informal movement/dance 
activities 

• Block play — construction play with blocks and other large 
building materials 

• Fine motor plav -manipulation of materials, such as puzzles, 
stringing beads, sewing cards, woodworking, LEGOs®, Lincoln 
Logs®, interconnecting building pieces. 

• Sensory play — manipulating sand, water, textured materials 
such as beans, rice, shaving cream, learning about qualities of 
materials rather than constructing a particular object. 

• Non-educational games with rules-p laving board games, card 
games that are not explicitly educational. 



6. Group / meeting time 

• Meeting time- routines or daily rituals as part of group or circle 
time. Includes repeated activities such as calendar, day of the 
week, weather, the day’s activities, etc. 



7. TV / video / computer 

Watching commercial television programs, videotapes / DVDs, or 
computer programs which may or may not be educational. Note: 
Circle “TV,” “Video,” or Computer” I to indicate which one child(ren) 
are using. 



8. Gross motor play 

Large muscle play — active outdoor play and indoor physical activity 
(tunnels, gymnastics). Include outdoor walks here (e.g., walking to 
and from a destination such as the library). 

Note: Code organized dance /movement activity as 

“Developmental activity. ” 



9. Physical care/self-help 

Bathroom, changing diapers, washing hands; includes comforting. 
Also includes helping children learn self-help skills such as 
putting on clothes, tying shoes. 

Note: Self-help involving eating goes under Meals/Snacks. 



10. Meals /Snacks 

Activities involved in the act of eating a meal /snack, and/or meal/ 
snack preparation and clean-up. 



1 1 . Transition/Routines / Management 

• Routines /transitions: Arriving /departing; transition between 
activities. 

• Conversation /management-Any talking or interaction 
between adult and child, between children, or between 
adults outside of a listed activity . Conversation may be 
positive or negative. Examples: adult managing a child’s 
behavior, comforting a child, or chatting. Children interacting in 
nonproductive ways would be coded in this “activity”. 



12. Not involved in any other listed activity 

Children roaming aimlessly around room, having a tantrum, solitary 
& note engaged in another listed activity; otherwise unengaged. 

Not interacting with another child or adult. 



13. In restraints 

Indicate number of infants or toddlers awake but in restraints 
(walkers, cribs, swings, high chairs) who are not included in other 
children's activities. 



14. Sleeping 

Indicate number of infants, toddlers or other children sleeping, 
regardless of where in the home they are asleep 



Provider or Assistant Language: 

If provider/assistant in activity is talking with child(ren), then write 
letter of language spoken by adult (E, S, or O). If the language 
used by the adult is rich, then circle the letter. 



Provider or Assistant Involvement: 

V — if provider or assistant is actively involved in teaching, playing, 
demonstrating, or discussion; 

L — if words, letters, or numbers used/present in activity 




















Read-Aloud Profile - R 
Fall 2007 



(RAP-R) 




Title of Book: 



Start Time : am/pm End time : am/pm 

A. Instructional Strategies Used by Reader 

( mark tally each time strategy is observed) 

1 Comprehension supports 

a Introduces background information related to the story/book 
b Relates text to class activities; reminds children of same/similar book read before 
c Narrates/tells the story in advance of reading 
d Talks about events and/or features to listen, look for in the story/book 
e Asks questions about book/connects to children’s personal experiences 
f Points to picture; names, talks about picture; asks questions about picture 
g Invites child to touch/point to picture 
h Question with known answer: who, what, when, where 

1 Summarizes story; retells story; acts out story/asks child to act out story 

j Non-discussion extension activity(art project/drama project based on story) 

2 Open-ended questions 

a Prediction (what’s going to happen in story); analysis (Why? question, inc feelings) 
b Hypothesize about real world or about things outside children’s experience 

3 Conventions of print 

a Tracks print 

b Discusses print conventions (punctuation, capitals, space between words) 
c Points to/discusses book features such as the title, illustrator, author 

4 Print knowledge 

a Comments on sounds, letters, sound/letter links; tells C to listen/look for them 
b Asks child to point to word(s), letter(s), or number(s) on page 

5 Print motivation 

a Uses props/dramatic voices/gestures' expresses enthusiasm for reading 
b Children join in reading/complete text on own as group (choral reading) 



6 Introduction of new story-related vocabulary 



RAP-R 



B. Frequency of 
Strategy 










C. Number of D. Reader’! 

Children, by Age Language 

(enter # tor each age) (circle all that 

apply) 



1 English 



Infants 

Toddlers 

Preschool 




D. Reader’s E. Features of Book 
Language (circle one response for each) 



2 Spanish 

3 Other 



B = book picture 
O = other picture 
P = prop: D = define 
S = semantic network 



El Type of book 



a Picture book 
b Alphabet book 

c Counting book 

d Chapter book 
e Reference book 
f Conversation book 



E2 Words/page 



a 0 words 

b 1 word 

c 2-10 words 

d >10 words 



E3 Language of book 



a English 
b Spanish 
c English & Spanish 
d Other language 



E4 Book features 



a Rhyming 

b Predictable/repetitive 



d Neither 

□ Reader reads book straight through with no 
codes in Column A. 

□ Read-aloud ends before book is finished. 

□ Audiotape or CD of book used. 



























Child/Provider Conversation Record 



Fall 2007 




Child/Provider Conversation Record 

Fall 2007 Child# 

Child Aqe vears/months 3 

Provider ID 

□ No talking occurred between provider or assistant and focal child during interval 


A 

Conver- 

sation 


B 

Number of Times Child/Provider 
Talks during Conversation 


C 

Topic/Activity Context 


Child Turns 


Provider Turns 


1 








2 








3 








4 








5 








6 








7 








8 








9 








10 








3 Indicate age in years for children 3 years and older, age in months for children under 3 
b For Topic/Activity, describe briefly and identify those that are management-oriented 



In table below, check the number of each type of conversation recorded above: 



# conversations 


0 


1 


2 


>2 


Management or helping 










Provider talk only (with no child response; 
includes provider narrating actions) 










Simultaneous talk (includes singing, chanting, 
reciting rhymes) 










Discussion (short or not building*) 










Extended discussion (4+ turns & building*) 





















* “Building” = provider’s turn builds on or extends child’s comment/response 






Arnett Caregiver Global Rating Scale 
With Additional Items from the QUEST Caregiver Rating Scale 



Summer 2006 
Fall 2007 
Fall/Winter 2008 1 



1 The Arnett was used at baseline (summer 2006), Fall 2007, and Fall/Winter 2008. The additional items 
from the QUEST Caregiver Rating Scale (Items 31-39) were collected Fall/Winter 2008. 




Arnett Global Rating Scale of 
Caregiver Behavior 


Not at all 


Somewhat 


Quite a bit 


Very much 


Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 


1 


Speaks warmly to children (e.g., positive 
tone of voice, body language). 


1 


2 


3 


4 

[warm] 


2 


Seems critical of children (e.g., puts 
children down, uses sarcasm). 


1 


2 


3 


4 

[critical 


3 


Listens attentively when children speak 

to her (e.g., looks at children, nods, 
rephrases their comments, engages in 
conversations). 


1 


2 


3 


4 

[listens] 


4 


Places high value on obedience (e.g., 
expects children to follow adult agenda, fails 
to respond to daily events in a flexible way). 


1 


2 


3 


4 

[obey] 


5 


Seems distant or detached from children 

(e.g., sits apart, does not touch children, 
does not greet children). 


1 


2 


3 


4 

[distant] 



Seems to enjoy children (e.g., conveys 



g warmth by smiling, touching, taking 12 3 4 

children’s conversations seriously). [enjoys] 

Reasons with children when they 

7 misbehave (e.g., explains rule they are 1 2 3 

breaking, discusses consequences, 
redirects behavior, discusses alternatives). 



8 


Encourages children to try new 
experiences (e.g., suggests children do it 
together, helps children start, introduces 
new materials). 


1 


2 


3 


4 

[try] 


9 


Exercises too much control over 
children (e.g., doesn’t seek child input, 
rigidly adheres to rules and schedules). 


1 


2 


3 


4 

[control] 


10 


Speaks with irritation or hostility to 
children (e.g., sharp tone, raises voice). 


1 


2 


3 


4 

[irritated] 



4 

[reasons] 



Definitions 



1 Rarely speaks warmly to children; neutral or negative 

2 Sometimes speaks warmly to children or only speaks warmly to some children 

3 Speaks warmly to children much of the time or to some children 

4 Consistently speaks warmly to children 

1 Never seems critical of children, verbally or non-verbally 

2 Rarely seems critical of children, verbally or non-verbally 

3 Often seems critical of children, verbally or non-verbally 

4 Consistently Is critical of or sarcastic with children, verbally or non-verbally 

1 Does not listen attentively when children speak 

2 Rarely listens attentively when children speak 

3 Usually listens attentively when children speak 

4 Consistently listens attentively when children speak — looks at children, talks with them 

1 Does not seem to place a high value on obedience Responds to daily events flexibly 

2 Sometimes expects children to follow rules/ conventions but allows some flexibility 

3 Usually emphasizes following rules or conventions rather than responding flexibly 

4 Consistently expects children to follow adult rules/conventions Does not respond to daily 

events flexibly 

1 Consistently is involved with, watches children with interest 

2 Is usually involved with children, but in some instances is distant or detached 

3 Is usually distant or detached but not always 

4 Is consistently distant or detached Sits apart, does not touch /talk to children 

1 Consistently fails to show enjoyment of/warmth toward children-may be harsh or distant 

2 Usually fails to show enjoyment of/warmth toward children 

3 Usually seems to enjoy children but sometimes is distant or harsh 

4 Consistently seems to enjoy children — warm, smiling, touching, listening seriously 

1 Rarely explains rules/discusses consequences when children break them 

2 Usually does riot explain rules, discuss consequences 

3 Usually explains the reasons for rules and consequences for breaking them 

4 Consistently explains the reasons for rules, explains consequences, redirects behavior 

1 Does not encourage children to try new experiences 

2 Rarely encourages children to try new experiences 

3 Often encourages children to try new experiences 

4 Consistently encourages children to try new experiences-provides opportunities/support 

1 Consistently seeks child input, does not rigidly adhere to rules and schedules 

2 Imposes some control over children’s behavior but usually allows children flexibility 

3 Keeps children in control most of the time-imposes rules and manages children's behavior 
closely 

4 Exercises too much control over children Adheres rigidly to rules/schedules with no child input 

1 Does not speak with irritation or hostility to children; does not uses sharp tone or raise voice 

2 Rarely speaks with irritation or hostility to children 

3 Often speaks with irritation or hostility to children 

4 Consistently speaks with irritation and/or hostility to children; uses sharp tone, raises voice 





Definitions 



1 Does not seems enthusiastic about children’s activities/efforts 

2 Rarely seems enthusiastic about children’s activities/efforts 

3 Usually seems enthusiastic about children’s activities/efforts 

4 Consistently seems enthusiastic about children’s activities/efforts; congratulates them, 

recognizes their efforts 

1 Does not threaten punishment/bribe children to control/manage them 

2 Usually does not bribe/threaten punishment to control children 

3 Often uses bribes/threats of punishment to control children 

4 Consistently uses bribes/threats of punishment to control children 

1 Rare spends time in activities that do not involve children 

2 Usually remains involved with children during 

3 Frequently spends time in activities that do not include children 

4 Spends substantial time in activities that do not include children 

1 Does not pay positive attention to children as individuals; does not single out children 

2 Rarely pays positive attention to children as individuals 

3 Often pays positive attention to children as individuals or pays attention to some but not all 
children 

4 Consistently pays positive attention to most/all children as individuals 

1 Rarely/never reprimands children too strongly Always tries to redirect children’s behavior 
without punishing them 

2 Sometimes reprimands children too strongly but sometimes deals with misbehavior by talking 
to children, offering alternatives, asking about reasons 

3 Often reprimands children who misbehave too strongly 

4 Consistently reprimands children too strongly-uses punishment, reprimands, threats 

1 Consistently talks to children in a way that is inappropriate for their developmental level 

2 Usually talks to children on a level they cannot understand 

3 Usually talks to children on a level they can understand, altering language for different levels 

4 Consistently talks to children at a level they can understand, checks to be sure they 

understand 

1 Does not punish children without explanation-discusses misbehavior, reason for rules 

2 Usually provides explanation for punishment and discusses misbehavior, rule 

3 Usually fails to provide explanations to children for punishment 

4 Consistently punishes children without explanation for punishment, discussion 

1 Consistently lets misbehavior, conflicts get out of control without exercising firmness-does not 
provide clear direction 

2 Often lets misbehavior, conflicts get out of control without stepping in 

3 Usually does not let misbehavior/conflicts get out of control without stepping in, exercising 
firmness 

4 Consistently exercises firmness, direction when necessary — doesn’t let things get out of 
control 




Arnett Global Rating Scale of 
Caregiver Behavior 



cc 



Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 


19 


Encourages children to exhibit prosocial 
behavior (e.g., sharing, cooperating, pairs 
socially skillful with those children who need 
practice). 


1 


2 


3 


4 

[share] 


20 


Finds fault easily with children (e.g., uses 
negative tone, is critical). 


1 


2 


3 


4 

[fault] 



21 


Fails to show interest in children’s 
activities (e.g., removes self from children’s 
activities, doesn't talk to children or extend 
their conversations). 


1 


2 


3 


4 

[no 

interest] 


22 


Seems to prohibit many of the things 
children want to do (e.g., adheres to rigid 
schedule or adult outcomes and agendas). 


1 


2 


3 


4 

[prohibit] 


23 


Fails to supervise children very closely 

(e.g., fails to foresee and forestall mishaps). 


1 


2 


3 


4 

[not 

close] 


24 


Expects children to exercise self-control 

(expects children to be undisruptive for 
short group, caregiver-let activities; to be 
able to stand in line calmly; reminds children 
of expectations, asks for cooperation in 
supportive ways). 


1 


2 


3 


4 

[self- 

control] 


25 


When talking to children, kneels, bends 
or sits at their level to establish better 
eye contact (e.g., e.g., ensures connection 
when having a conversation). 


1 


2 


3 


4 

[kneels] 


26 


Seems unnecessarily harsh when 
scolding or prohibiting children (e.g., 
angry tone, shakes children, uses physical 
punishment, uses “time out” without 
explanation). 


1 


2 


3 


4 

[harsh] 



C\J CO xf- 



Definitions 



1 Does not encourage prosocial behavior (sharing, cooperation) 

2 Rarely encourages prosocial behavior 

3 Usually encourages prosocial behavior 

4 Consistently takes advantage of opportunities to encourage prosocial behavior; encourages 
children to work together, help each other 

1 Consistently avoids finding fault easily with children; does not blame or criticize needlessly 

2 Rarely finds fault easily with children, blames or criticizes needlessly 

3 Often finds fault with children too easily, blames or criticizes needlessly 

4 Consistently finds fault too easily with children Is quick to criticize, lay blame, 

1 Consistently shows interest in children’s activities, talking to/interacting with them 

2 Often shows interest in children's activities but sometimes does not or not for some children 

3 Often fails to show interest in children’s activities or interacts with them in activities, 

4 Consistently fails to show interest in children’s activities or to interact with them in activities 

1 Does not prohibit most things children want to do — consistently lets them decide what to do 

2 Rarely prohibits what children want to do; usually lets them decide 

3 Often prohibits what children want to do 

4 Consistently prohibits what children want to do Follows adult agenda rigidly 

1 Consistently supervises closely, rarely letting children out of sight or hearing; foresees 

mishaps 

Usually supervises children closely; usually forestalls and foresees mishaps 
Often fails to supervise children very closely Often fails to foresee mishaps 
Consistently fails to supervise very closely, often leaving children out of sight or hearing; fails 
to foresee mishaps 

1 Does not expect children to control their own behavior-to be able to stand in line, wait their 
turn, listen and participate in an orderly way during caregiver-led activities 

2 Usually does not expect children to be in control of their own behavior 

3 Usually expects children to be in control of their own behavior 

4 Expects children to exercise self-control; asks for cooperation in supportive ways 



1 Does not kneel, bend when talking to children; does not ensure a connection 

2 Rarely kneels, bends when talking to children 

3 Usually kneels, bends when talking to children to ensure a connection 

4 Consistently moves to child’s height when talking to ensure a connection; establishes eye 

contact 

1 Consistently avoids unnecessary harshness in discipline; does not use angry tone, use 
physical punishment, use “time out” without explanation 

2 Usually avoids unnecessary harshness when disciplining children 

3 Often seems unnecessarily harsh when scolding children 

4 Consistently seems unnecessarily harsh when scolding or prohibiting; uses angry tone, 

physical punishment, “time out” without explanation 





Arnett Global Rating Scale of 
Caregiver Behavior 

Circle one number for each item that best represents the extent are 



each of the following statements characteristic of this caregiver: 



27 


Does self-help tasks for children (e.g., 
dresses them, wipes their noses, selects 
materials for choice time). 


1 


2 


3 


4 

[does 

tasks] 


28 


Does not get child assistance with tasks 

(e.g., watering plants, animal care, putting 
away toys, cleaning tables and other routine 
maintenance). 


1 


2 


3 


4 

[no 

assist] 


29 


Promotes leadership skills (e.g, gives 
assignments for jobs like line leaders, clean- 
up inspector, talks about children’s 
contributions to the group). 


1 


2 


3 


4 

[leaders] 



Assists children in making productive 
30 choices (e.g., uses a planning process or 1 2 3 

discussion to outline choices during activity 
periods, narrows choices when children 
have difficulty). 




4 

[choices] 



Definitions 



1 Consistently lets children do own self-help tasks-wipe own noises, wash own hands, dress for 
outdoors, select materials for choice time. 

2 Often lets children do own self-help tasks. 

3 Often does self-help tasks for children. 

4 Consistently does self-help tasks for children 

1 Consistently gets children to help with tasks 

2 Usually gets children to help with tasks 

3 Rarely gets children to help with tasks 

4 Does not get children to help with tasks-water plans, care for pets, put away toys/materials 

1 Does not promote leadership skills in children-does not assign jobs, talk about children’s 
contributions to group. 

2 Rarely promotes leadership skills in children 

3 Often promotes leadership skills in children. 

4 Consistently promotes children’s leadership skills through job assignments, talk about 
contributions to group, asking children’s opinions 

1 Does not assist children in making productive choices by discussing choices, helping narrow 
choices, redirecting poor choices 

2 Usually does not assist children in making productive choice. 

3 Usually assists children in making productive choices. 

4 Consistently assists children in making productive choices. 




Arnett Global Rating Scale of 
Caregiver Behavior 




Definitions 



1 Never has extended interactions with or conversations with a single child or a pair of children; 
either has only brief interactions or interacts with children only in groups of 3 or more. 

2 Infrequently (1 time) has extended interactions or conversations with a single child or a pair of 
children. 

3 Occasionally (2-3 times) has extended interactions or conversations with a single child or a pair 
of children. 

4 Often (4 or more times) has extended interactions or conversations with a single child or a pair 

of children. 

1 Never nudges children to try something new or add to an on-going activity. 

2 Infrequently (1 time) nudges children to try something new or add to an on-going activity. 

3 Occasionally (2-3 times) nudges children to try something new or add to an on-going activity. 

4 Often (4 or more times) nudges children to try something new or add to an on-going activity. 

1 Never enriches routines or transitions with language interactions or learning opportunities; is 
either occupied elsewhere during routines or interactions with children are focused on 
managing behavior. 

2 Infrequently (1 time) enriches routines or transitions with language interactions or learning 
opportunities. 

3 Occasionally (2-3 times) enriches routines or transitions with language interactions or learning 
opportunities. 

4 Often (4 or more times) enriches routines or transitions by conversing with children, singing or 

reciting rhymes or poems, or creating learning opportunities. 

1 Never engages in language-rich interactions with children; most language interactions with 
children are limited to behavior management. 

2 Infrequently (1 time) engages in language-rich interactions with children. 

3 Occasionally (2-3 times) engages in language-rich interactions with children. 

4 Often (4 or more times) engages in language-rich interactions with children. 



1 Never encourages children to look at or read books or other print materials on their own; books 
or other print materials are never available. 

2 Infrequently (1 time) encourages children to look at or read books or other print materials on 
their own; books or other print materials are infrequently available. 

3 Occasionally (2-3 times) encourages children to look at or read books or other print materials 
on their own; books or other print materials are occasionally available. 

4 Often (4 or more times) encourages children to look at or read books or other print materials on 
their own; books or other print materials are available to children a large portion of the time. 

1 Never uses open-ended questions with children 3 years and older. 

2 Infrequently (1 time) uses open-ended questions with children 3 years and older. 

3 Occasionally (2-3 times) uses open-ended questions with children 3 years and older. 

4 Often (4 or more times) uses open-ended questions with children 3 years and older. 

NA= Not applicable; there are no children 3 years old or older present. 
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Arnett Global Rating Scale of 
Caregiver Behavior 



Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 

37 Provides a variety of activities for 

children to engage in on their own or 12 3 

materials that support children’s own I v 

activities (e.g., children usually have 
activities or materials that they can use on 
their own without the provider intervening, 
interacting with them, providing direction, or 
offering alternative materials or activities). 

38 Encourages infants (children who can 

not walk on their own) to explore and be 1 2 3 

active (e.g., encourages infants to move I enc< 

around, get involved in activities, explore, 
and play with materials by providing safe 
and age-appropriate materials, activities, 
and equipment, and by providing them with 
time and space for supervised self-directed 
movement) If an infant is in restraints (e.g. a 
high chair, car seat, or swing) for more than 
20 minutes at one time, code '1 '. 



4 

[variety] 



[encourages] 



Very much 



Definitions 



1 Never provides activities or materials that children can engage with; children are usually bored 
or uninvolved without provider intervention. 

2 Infrequently (1 time) provides activities or materials that children can engage with; children are 
frequently bored or uninvolved without provider intervention. 

3 Occasionally (2-3 times) provides activities or materials that children can engage with; children 
are infrequently bored or uninvolved without provider intervention. 

4 Often (4 or more times) provides activities or materials that children can engage with; children 
are never bored or uninvolved without provider intervention. 

1 Never encourages infants to explore and be active OR keeps an infant in restraints (e.g. a high 
chair, car seat, swing) for more than 20 minutes at one time. 

2 Infrequently (1 time) encourages infants to explore and be active OR does so for less than 20 
minutes. 

3 Occasionally (2-3 times) encourages infants to explore and be active OR does so for more than 
20 minutes but less than an hour. 

4 Often (4 or more times) encourages infants to explore and be active OR does so for an hour or 
more. 

NA= Not applicable; there are no infants present. 




Snapshot of Activities: Family Child Care Home (OMLIT-SNAP) 



Fall 2006 




Snapshot of Activities: Family Child Care Home (OMLIT-SNAP) 



Number 


Children Present in Home 




Infants & toddlers (under 3 yrs old) 




Preschool (3-5 yrs old) 




School age (5+ yrs old) 




TOTAL Children (all ages) 



Number 


Staff 




Provider 




Assistant (paid) 


Number 


Other Adults Present in Home 




Adult dropping off/picking up/visiting 




Volunteer / visitor (non-relative) 




Other family member 




TOTAL Adults (staff and other adults) 



Snapshot #1 

Time: : am pm 



Key: 

Language(s) spoken: 

E = English 
S = Spanish 
O = Other 

*=Provider’s own child (on roster) 



Activity 






















Care- 

giver 


Other 

Adults 


i- Literacy 


Description of 
Activity 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


' 1 


Reading /text /vocabulary (w /print) 


























L 




la 


Alphabet /numerals 


























L 




1b 


Sounds/singing 






























1c 


Oral language/vocabulary (no print) 






























2 


Emergent writing /copying /tracing 


























L 




3 


Science /nature 






























4 


Math concepts /attributes /colors 






























5 


Dramatic play 






























6 


Creative play 






























7 


Block play 






























8 


Fine motor play 






























9 


Sensory play 






























10 


Meeting time (daily routines) 






























11 


Games with rules 






























12 


TV /video /computer 






























13 


Gross motor play 






























14 


Other activity 






























15 


Routines /management 






























16 


Meals /snacks 






























17 


Napping 






























18 


Child not involved in activity 











































SNAPSHOT Definitions 



1. Reading /text /vocabulary ( with print) 

Looking at books or pictures, adult reading aloud, children 
reading together without adult, emergent reading (pretending to 
read), shared reading activities. Listening to stories on audiotape 
or CD. Teaching children new vocabulary words with print 
support for vocabulary (e.g., printed word). 

la. Alphabet /numerals 

Recognizing letter/numeral forms, letter /sound correspondence. 

Always involves print. 

lb. Sounds /singing 

Sounds of words with no print. All singing (may or may not have 
print, e.g., words of song displayed). 

lc. Oral language /vocabulary (no print) 

Discussions, new concepts and vocabulary with eitherno print 
or no print emphasis. 

2. Emergent writing /copying /tracing 

Child(ren) writing, includes pretend writing, scribbling, invented 
spelling. Child dictation to teacher. Tracing letter or number 
templates. Practice in correctly writing numerals /distinguishing 
numerals. Always involves print 

3. Science /nature 

Formal and informal communication of science or nature. 

Science examples: astronomy, working with pets , collecting 
leaves, feeding pets, magnets, health & safety. 

4. Math concepts /attributes /colors 

Formal and informal communication of math concepts, attributes, 
or colors. Shapes, counting, measuring, patterns, amount. 
Identifying and matching non-geometric shapes (animals, 
familiar objects). Identifying and matching colors and color 
names. 

5. Dramatic play 

Pretend or make-believe play; dress-up, playing with dolls; 
assigning roles; zooming cars and trucks. 

Note: Includes acting out stories /playing with puppets, figures of 
people/animals, and stuffed animals in pretend environments. 

6. Creative play 

Arts and crafts - creating visual art (painting, drawing, sculpting 
clay & play dough, cutting and pasting). Note: Always code 
PlayDoh® as “ Creative play.” 

Music - instruments, formal and informal movement/dance 
activities. 

7. Block play 

All building with blocks and other large building materials. 

Note: Once construction is done, and blocks are part of a 
completed pretend environment with cars, trucks, figures of 
people, code as “ Dramatic play”. Fine motor play 

Manipulation of materials, such as puzzles, stringing beads, 
sewing cards, woodworking, LEGOs®, Lincoln Logs®, 
interconnecting building pieces. 



9. Sensory play 

Manipulating sand, water, and textured materials such as beans, 
rice, shaving cream, where objective is learning about qualities 
of materials and not constructing a particular object. 

10. Meeting time 

Routines or daily rituals as part of group or circle time. Includes 
activities such as calendar, day of the week, weather, the day’s 
activities, etc. 

11. Games with rules 

Playing board games, card games, and video games (e.g., 
Nintendo, Game Boy, Play Station) that are not explicitly 
educational. 

12. TV /video /computer 

Watching commercial television programs, video tapes / DVDs, 
or computer programs which may or may not be educational. 
Note: If activity involves computer, circle “ Computer ” in 
description box. 

13. Gross motor play 

Large muscle play — active outdoor play and indoor physical 
activity (tunnels, gymnastics). Include outdoor walks here (e.g., 
walking to and from a destination such as the library). 

Note: Code organized dance /movement activity as “Creative 
play." 

14. Other activity 

Special activities that are not part of the regular activities on list, 
e.g. special events /destinations, field trips, student assessments 
school assembly, library). Note: Specify activity in description 
box. 

15. Routines / Management 

Routines /transitions: Arriving /departing, napping /sleeping, 
physical care/hygiene (including first aid, toileting), setting up or 
cleaning up of activities /materials, lining up. 

Conversation /management: Any talking or interaction between 
adult and child, between children, or between adults outside of a 
listed activity . Conversation may be positive or negative. 
Examples: adult managing a child’s behavior, comforting a child, 
or chatting. Children may be interacting in nonproductive ways. 

Uninvolved /administration: Adults not involved in any activity 
listed above and not interacting with another person . Teacher/ 
other adult doing administrative work, monitoring overall 
classroom activity from a distance. 

1 6. Meals /Snacks 



Engaged in the act of eating a meal /snack, and/or meal /snack 
preparation and clean-up. 




Child roaming aimlessly around classroom, having a tantrum, 
otherwise unengaged. 























Read-Aloud Profile (OMLIT-RAP) 

Adapted for the Massachusetts Family Child Care Study 

Fall 2006 




A. PRE-Reading 

1. Guides book choice; discusses children’s book choice(s): 

Adult encourages children to choose the book; talks about 
their choice with them. Helps them make appropriate choice. 

2. Points to features of the book such as the title, 

illustrations, author: Points to title, author, illustrator, or 
illustration on front of book (or points to chapter title in a 
chapter book) 

2a. Discusses concepts of print such as the title, illustrator, 
author: Defines, describe meaning of concepts of print such 
as title, author, illustrator, or illustration. 

3. Reminds children of similar books they have read or that 

they have read same book before: Calls attention to books 
children by the same author, same illustrator, same topic, etc. 
OR reminds children they’ve already read same book before. 
Ex: “What was another book that we read about ducks?” or “ 
...by Eric Carle?” 

4. Comments on sounds, letters, and/or sound-letter links to 

listen, look for in the story/book: Talks about sounds they 
will hear in the story, especially sounds they may have been 
learning about in class. Or talks about letters they will see in 
the story, especially letters they have been learning about. 

Ex: “During the story, when you hear the ‘buh’ sound, raise 
your hand.” Or “This story has a lot of words that begin with 
the letter ‘g.’ Let me know when you see one.” 

5. Introduces story-related vocabulary: Highlights or explains 

new vocabulary. Ex: “This book is about a fish called a 
‘sunfish.’ Sunfish have fins. Fins are what they use to move 
around in the water. When we read the book, you will see 
pictures of sunfish and we can pick out their fins.” 

6. Relates the story/book to other activities in class, class 

theme: Calls attention to the book’s relation to class 
activities or theme. Ex: “This duck likes to eat fish. What does 
our pet turtle like to eat?” Or, “Remember last week when we 
went to the fire station? This story is about firemen like the 
ones we met.” 

7. Talks about events and/or features to listen, look for in the 

story/book: Helps children anticipate things that will happen 
in the book. Ex: “At the end well talk about all the different 
things that the caterpillar likes to eat. What do we think his 
favorite food is?” 

8. Introduces background information related to the 

story/book (with or without child input): Describes what 
the book is about. Ex: “This book is about a birthday party 
that Little Bear has with all his forest friends.” May or may not 
invite child discussion. 

9. Narrates/tells the story in advance of reading: Recites all or 

major parts of book (e.g., nursery rhyme in book based on 
rhyme) before actually reading story. 

10. Relates the story/book to children’s own experiences 

outside of classroom activities: Links book to children's 
experiences outside of class. For example: “Have you ever 
fed the ducks in the park before? What sound did they 
make?” or “What kinds of things do you like to do on a rainy 
day?” 

11. No pre-reading experiences or activities (without any 

codes 1-10): The adult may alert children to the reading 
activity but does not provide any of the above listed 
experiences/activities. 



B. Reading 

1. Tracks print: Adult moves her finger along the page below 

the line of print or points sequentially to words in text as she 
reads. 

2. Uses props / dramatic voices / gestures: Uses props (e.g., 

hand puppets, stuffed animals, items in the story), gestures, 
or different voices to tell story. 

3. Directs children’s attention to illustrations/text/story. 

(i.e., asks questions about; discusses/expands on 

meaning; offers new information): Points to, or in some 
way calls children’s attention to the book’s illustrations, details 
of the illustrations. Engages children in (brief) discussion 
about the meaning of text or illustrations and/or offers new 
information about the story that may not be written in the text 
or depicted in illustration (e.g., explaining about something 
unfamiliar in the text or pictures). 

4. Comments on sounds, letters, and/or sound-letter links in 

the story/book: Calls children’s attention to sounds, letters, 
or sounds and their corresponding letters in the story text. 

5. Highlights new story-related vocabulary: Calls attention to, 

defines, and/or gives examples to help children understand 
unfamiliar words in story. 

6. Relates the story/book to other activities in class, class 

theme: Places the story in context by mentioning the class 
theme and/or how the book fits into the class activities. 

7. Expands on children’s comments about the story/book: 

When child makes a comment, adult extends by asking child 
for elaboration or restating child’s comment. 

8. Answers children’s questions about the story/book or 

related topics: Allows children to ask questions about the 
story and then responds to those questions. 

9. Has children join in reading/completing text on their own or 

as a group (choral reading): Pauses and/or indicates to 
children in some way that they should recite words/numbers, 
phrases, or longer chunks of the text aloud with her. 

10. Asks recall questions about earlier parts of the 
story/book: Asks children to recall events, characters, 
attributes from earlier in the story. 

11. Relates the story/book to children’s experiences/ Asks 
the story/book-related questions about children’s 
experiences outside of classroom activities: Extends 
children’s understanding by tapping into their own 
experiences to help them comprehend the story. 

12. Asks the story/book-related open-ended questions 
(requires prediction, expanded response, thinking, and/or 
analysis): Probes children's comprehension by asking 
questions about the story that require children to predict (e.g., 
“What do you think will happen next? What if...”), elaborate 
responses; engage in more thought or analysis of the story. 

13. Picture walk: “Walks through” the book without reading text; 
turns pages and describes aspects of the illustrations, and/or 
asks children about the illustrations. May or may not “tell” the 
story. 

14. Reads text straight through (without any codes 1-13): 

The adult does not engage children in any of the activities or 
behaviors listed above while reading the story. 



C. POST-Reading 

1. Answers children’s questions about the story/book or 

related topics: Teacher reads, speaks, without expecting 
response from children at that time. 

2. Expands on children’s comments about the story/book or 

illustrations: Allows children to ask questions about the 
story and then responds to those questions. 

3. Comments on sounds, letters, and/or sound-letter links in 

the story/book: Calls children’s attention to sounds, letters, 
or sounds and their corresponding letters in the story text. 

4. Reviews/reinforces story-related vocabulary with or 

without print reference: The teacher suggests ways of 
extending activities (e.g., to include or extend literacy activity) 
or offers materials (e.g., literacy materials). 

5. Summarizes/retells the story without child involvement: 

Re-tells plot of story to remind children; help children who 
didn’t understand what the story meant. 

6. Summarizes/retells the story with child involvement: 

Involves children in retelling plot of story. 

7. Asks for recall of information about the story/book: Asks 

children to recall events, characters, attributes from the story 
she just read aloud. 

8. Asks story/book-related questions about children’s 

experiences outside of classroom activities: Extends 
children’s understanding by tapping into their own 
experiences to help them comprehend the story. 

9. Asks story/book-related open-ended questions (requires 

speculation, expanded response, thinking, and/or 
analysis): Probes children’s comprehension by asking 
questions about the story that require speculation, longer or 
more elaborated responses, more thought, or analysis of the 
story. Ex: “What do you think would have happened if...?” 

10. Organizes post-reading story/book -related activity 
(beyond oral discussion): The teacher suggests ways of 
extending activities (e.g., to include or extend literacy activity) 
or offers materials (e.g., literacy materials). 

11. No post-reading activity or extension occurs (without any 
codes 1-10): The adult does none of the above listed 
extensions or activities after reading the book. 

Note: Discussion of concepts of print during post-reading should 
be coded as A-2a. 







Read-Aloud Profile 



©MLIT-RAP 



Start Time 


am pm 


Title of Book: 


Staff ID# RAP #1 


End Time 


am pm 


Author: 


□ Read-Aloud ends before book is completed 



A. PRE-Reading 
(set-up) 

(circle all that apply) 


B. Reading 

(circle all that apply) 


C. POST-Reading 
(extension) 

(circle all that apply) 


D. Adult 
Reading 
Book 

(circle all that 
apply) 


E. Adult 
Language 
with 

Children 

(circle all that 
apply) 


F. Number of 
Children 
Reading 

(circle one of 
FI -4; also 
F5 if applies) 


G. Book 

Characteristics 

(circle one for each) 


Guides book choice; 

1 discusses children’s book 
choice(s) 


1 


Tracks print 


1 


Answers children’s questions 
about the story/book or related 
topics 


1 


Teacher 


1 


English 


1 


One 

child 


la 

1b 

1c 

Id 

1e 


Tvoe of book 
Picture book 
Alphabet book 
Counting book 
Chapter book 
Reference book 


2 


Points to features of the book 
such as the title, illustrations, 
author 


2 


Uses props/dramatic voices/gestures 


2 


Assistant/ 

Aide 


2 


Spanish 


2 


Two children 


2a 


Discusses/defines concepts 
of print such as the title, 
illustrator, author 


3 


Directs children’s attention to 
illustrations/text/story (i.e., asks 
questions about; discusses/expands 
on meaning; offers new information) 


2 


Expands on children’s 
comments about the story/book 
or illustrations 


3 


Other 

Adult 


3 


Other 

language 


3 


Small group 
(3-5 children) 


2a 

2b 


Bia Book 

Yes 

No 


3 


Reminds children of similar 
books they have read or that 
they have read same book 
before 


4 


Comments on sounds, letters, and/or 
sound-letter links in the story/book 


3 


Comments on sounds, letters, 
and/or sound-letter links in the 
story/book 


Vocabulary & Supports: 


4 


Large group 
(6+ children) 


3a 

3b 

3c 

3d 

3e 

□ 


Lanauaae 
English only 
Spanish only 
Eng & Spanish 
Other language 
Eng & other 
language 
Book is read in 
different language 


4 


Comments on sounds, 
letters, sound/letter links, or 
tells children to listen and 
look for them in story 


5 


Highlights new story-related 
vocabulary 


4 


Reviews/reinforces story-related 
vocabulary with or without print 
reference 


5 


Whole class 


6 


Relates the story/book to other 
activities in class, class theme 






5 


Introduces story-related 
vocabulary 


7 


Expands on children’s comments 
about the story/book 


5 


Summarizes/retells the story 
without child involvement 


Open-ended Questions: 


4a 

4b 

4c 

4d 


Words/Daae 

0 words 

1 word 
2-10 words 
>10 words 


6 


Relates the story/book to 
other activities in class, class 
theme 


8 


Answers children's questions about 
the story/book or related topics 


7 


Talks about events and/or 
features to listen, look for in 
the story/book 


9 


Has children join in reading/completing 
text on their own or as a group (choral 
reading) 


6 


Summarizes/retells the story 
with child involvement 


5a 

5b 


Book on tape 

Yes 

No 


8 


Introduces background 
information related to the 
story/book (book (with or 
without child input) 


10 


Asks recall questions about earlier 
parts of the story/book 


7 


Asks for recall of information 
about the story/book 


6a 

6b 

6c 


Related to class 
theme 

Yes 

No 

Don’t know 


9 


Narrates/tells the story in 
advance of reading 


11 


Relates the story/book to children’s 
experiences/Asks story/book-related 
questions about children’s experiences 
outside of classroom activities 


8 


Asks story/book-related 
questions about children’s 
experiences outside of 
classroom activities 


10 


Relates the story/book to 
children’s experiences 
outside of classroom 
activities 


12 


Asks storv-related open-ended 
questions (requires prediction, 
expanded response, thinking, and/or 
analysis) 


9 


Asks story/book-related open- 
ended auestions (reauires 
speculation, expanded 
response, thinking, analysis) 






11 


No pre-reading 
experiences or activities 
(without any codes 1-10) 


13 


Picture Walk 


10 


Organizes post-reading 
story/book-related activity 
(beyond oral discussion) 






14 


Reads text straight through 
(without any codes 1-13) 


11 


No post-reading activities or 
extension occurs 
(without any codes 1-10) 



A. PRE-Reading 
(set-up) 


B. 


Reading 




(circle all that apply) 




(circle all that apply) 


12 


Invites child to read. 


15 


Points to illustrations while naming them 
or talking about them [“see”]. 


13 


Takes child on lap or positions 
child close to herself. 


16 


Asks child to point to specific 
illustrations, [“show”]. 


14 


Asks rhetorical introductory 
questions. 


17 


Asks child to name specific illustrations 
[“say”]. 




18 


Invites child to act out a part of the story 
or act out answers to questions. 



Read-Aloud Profile 



9 MLIT-RAP 



A. Pre-Reading 



12. Invites child to read by talking to child and/or 

holding up book for child to see: Talks about 
book, e.g. “Here’s something that will be fun to 
read” and/or displays book so the child can see it 
as the adult approaches child. 

13. Takes child on lap or positions child close to 

herself: This assumes that there are two or 
three children at most in the reading session. 

14. Asks rhetorical introductory questions such 

as “I wonder what this story will be about”: 

Asks a question even though the child is too 
young to answer. May give the answer to her 
own question. 



B. Reading 



15. Points to illustrations while naming them or 

talking about them (to draw children’s attention 
to the book illustrations), [“see”] 

*can also be done without pointing (see manual) 

16. Asks child to point to specific illustrations. 

For example, “Point to the kitty.” Or “Show me 
the boy wearing a hat.” [“show”] 

*can be a variety of actions (see manual) 

17. Asks child to name specific illustrations: The 

adult asks for a verbal response from the child 
with questions such as, “Who’s this?” or “ What 
do you see on this page?” [“say”] 

18. Invites child to act out a part of the story or act 

out answers to questions: Asks a question that 
the child can answer with a gesture or 
movement, such as “How do birds use their 
wings?” 





Quality Indicators for OMLIT-RAP 



RAP #1 



1. Story- 

related 

vocabulary 

Code as “1 " if 
no A5, B5, or 
C3 is circled 
Code item as 
“1 ” if no new 
vocabulary 
introduced. 


□ 1 = Minimal 


□ 2 


□ 3 = Moderate 


□ 4 


□ 5 = High 


Some story-related vocabulary words 
are introduced/discussed, but the 
definition of one or more of the words 
is misleading or wrong. 




One story-related vocabulary word is 
introduced or discussed and the 
definition is accurate AND 
At least one of the followina 
comprehension supports is given for 
the word: 

• A picture, gesture, or other 
concrete visual aid is used; 

• The word is linked to a rich 
network of related words or 
concepts. 




At least 2 story-related vocabulary 
words are introduced or discussed and 
the definition of each vocabulary word 
is accurate AND 

Both of the following comprehension 
supports are given for each word: 

• A picture, gesture, or other concrete 
visual aid is used; and 

• Each word is linked to a rich network 
of related words or concepts. 


2. Adult use 
of open- 
ended 
questions 

Code item as 
"1 ” if no 
open-ended 
questions (no 
A10, Bit, 
B12, C8 or 
C9 circled ) 


□ 1 = Minimal 


□ 2 


□ 3 = Moderate 


□ 4 


□ 5 = High 


Adult poses only one open-ended 
question and does not provide 
opportunity for children to respond to 
question (child not given time to 
respond, or adult moves on after child 
has responded). 




Adult poses two open-ended 
questions and provides opportunity 
for children to respond to one but 
not both of the questions. 




Adult poses at least four open-ended 
questions and consistently shows 
interest in/actively encourages 
children’s responses (e.g., pausing for 
children, restating question, calling on 
particular children, acknowledging 
children’s response). 


3. Depth of 

Post- 

Reading 

Code item as 
“1” if no CI- 
CIO is 
circled) 


□ 1 = Minimal 


□ 2 


□ 3 = Moderate 


□ 4 


□ 5 = High 


No post-reading extension or 
activities. 

(Post-reading coded as C11) 


Relates to the book but 
lasts LESS than 5 
minutes 


Discussion and/or activity that 

• Relates to the story/book but 
does not extend its meaning or 
comprehension 

AND 

• Lasts at least 5 minutes. 


Extends comprehension 
and lasts LESS than 10 
minutes 


Discussion and/or activity that 

• Extends the meaning of the text and 
reinforces comprehension of the 
story/book 

AND 

• Lasts at least 1 0 minutes. 








Observations of Provider Interactions 
With Children 



Fall 2006 



1 




Provider’s Interactions With Children 



Observation# Start time: am/pm 


A. Child Activity Context 


Activity 


1 


2 


3 


4 


5 


1. P Interacts (with 1-2 children) during the following 
type of activity: (Check one) 

a. LearningGames (LG) activity 

b. Non-LG activity 

c. Free play 

d. Interactive reading 

e. Routine or transition 

f. Other 


Activity 

Code: 

□ a. 

□ b. 

□ c. 

□ d. 

□ e. 

□ f. 


Activity 

Code: 

□ a. 

□ b. 

□ c. 

□ d. 

□ e. 

□ f. 


Activity 

Code: 

□ a. 

□ b. 

□ c. 

□ d. 

□ e. 

□ f. 


Activity 

Code: 

□ a. 

□ b. 

□ c. 

□ d. 

□ e. 

□ f. 


Activity 

Code: 

□ a. 

□ b. 

□ c. 

□ d. 

□ e. 

□ f. 


2. Briefly describe activity/LearningGame 












3. Number of children involved in context activity 
(if more than 2) 












4. Age ranges of children involved - no more than 2 
total. (Write the number of children in each age 
range, up to 2 children in one age range or one in 
each of 2 age ranges) 


a. < 1 

b. 1-2 

c. 3-5 

d. 6+ 


a. < 1 

b. 1-2 

c. 3-5 

d. 6+ 


a. < 1 

b. 1-2 

c. 3-5 

d. 6+ 


a. < 1 

b. 1-2 

c. 3-5 

d. 6+ 


a. < 1 

b. 1-2 

c. 3-5 

d. 6+ 


B. Verbal Behaviors (Check all that apply) 


1 . P nudges C to try something new/extend activity 

• Makes suggestions 

• Asks questions 












2. P narrates/talks about what C is doing, or 

• Asks questions about what C is doing 

• Gives feedback on what C is doing 

Do not count giving directions or discipline. 












3. P introduces/extends language play or learning 
with Child (Ex: sings, plays sound/rhyming game, 
uses letters or print, teachers new word, plays back- 
and-forth game with nonverbal child) 












4. P encourages/supports Child to do activity on 

own (Ex: feed self, ride bike, use scissors, look at 
book) 












C. Non-Verbal Behaviors (Check all that apply) 


1 . P nudges Child to try something new/extend an 
activity: 

• Offers a new toy or material 












D. Interactive Book Reading Behaviors (Code only for Activity Aid. Check all that apply) \ 


1 . P reads/talks about book and points to pictures while 
Child is actively looking at book (“see”) 

• P does not explicitly ask Child for a response 












2. P asks Child question about book and waits for 
nonverbal response from Child (“show”) 












3. P asks Child questions about book and waits for 
verbal response from Child (“say”) 












E. Reasons for not Coding Any Behaviors (Check one if no other codes were checked during the observation period): 


1 . P interacted with 1 -2 children but none of the 
behaviors (B-D) were observed □ 












2. P only interacted with 3 or more children □ 




3. P did not interact with any C/out of room □ 


4. RAP □ 


5. Other (specify) □ 
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QUEST Caregiver Rating Scale 



Summer 2006 
Fall 2007 




Relationships and Learning Activities 


Not True/Rarely True 
Little/No Evidence 


Sometimes True 
Some Evidence 


Usually/Always True 
Consistent Evidence 


Not Applicable/ 
Not Observed 




Definitions 


Caregiver With Children--Caring and Responding 




















i 


Rarely/never pays attention to children/noticeable lack of interest in children 


1 


Caregiver shows interest in what children say 


1 


2 


3 






2 


Inconsistent attention/interest OR only attention to some children 




and do, and listens attentively to them. 












3 


Consistent attention to and interest in children* 
















1 


Rarely/never responsive to children’s language/vocalizations 


2 


Caregiver responds to children’s language and 


1 


2 


3 






2 


Only sometimes responsive to children 




babies’ vocalizations. 












3 


Consistently responsive verbally or non-verbally to most/all children* 
















1 


No/almost no individual caregiver/child in-depth conversation with most children 


3 


Caregiver has conversations with each child. 


1 


2 


3 






2 


Individual in-depth conversations with a few/some children 
















3 


Individual in-depth conversations with most/all children* 




Caregiver shows affection to each child, 












1 


Rarely/never shows affection to children 


4 


including gentle touch, kind words, special looks. 


1 


2 


3 






2 


Shows affection to some children but not to others 
















3 


Consistently shows affection to most/all children* 
















1 


Rarely/never notices/offers help, or help is interfering rather than helpful 


5 


Caregiver offers children help when needed. 


1 


2 


3 






2 


Sometimes notices and offers help OR only notices some children 
















3 


Consistently notices/offers help when needed/does not interfere/force help 
















1 


Rarely/never acknowledges children’s efforts 


6 


Caregiver acknowledges specific aspects of 


1 


2 


3 






2 


Only sometimes acknowledges children's efforts OR only acknowledges efforts of 




each child’s efforts and accomplishments. 














some children 
















3 


Consistently acknowledges each child s efforts 
















1 


Rarely/never accepts children’s feelings 


7 


Caregiver accepts children’s feelings. 


1 


2 


3 






2 


Sometimes accepts children’s feelings/or only feelings of some children 
















3 


Consistently accepts children's feelings/attends to how most/all children feel* 
















1 


Rarely/never recognizes signs of distress/doesn’t respond appropriately 


8 


Caregiver recognizes signs of distress in 


1 


2 


3 






2 


Sometimes recognizes/responds to signs of distress or only responds to some 




children and responds and comforts them. 














children 
















3 


Consistently recognizes/responds appropriately to distress for most/all children* 
















1 


No interactive play with children 


9 


Caregiver plays interactively with children. 


1 


2 


3 






2 


Plays interactively 1 -2 times, OR only with some children 
















3 


Plays interactively more than two times with all/most children* 
















1 


Rarely/never holds babies for any reason 


10 


Caregiver holds babies. 


1 


2 


3 


C 

2 C\| 




2 


Holds babies only some of the time/only some babies/only holds babies in distress 












-O 5- 
= CD 




3 


All/most babies are held both when in distress and other times 












o ? 
O =5 
Z 






Code as NA if no children under 2 years old. 



'Note: In setting with 4 or fewer children, code 3 only if caregiver does this with all children; 



in setting with 5 children, code 3 if caregiver does this with at least 4 of 5 children: 

in setting with 6 children, code 3 if caregiver does this with at least 4 of 6 children: 

in setting with 7 or 8 children, code 3 if caregiver does this with at least 5 children. 




Definitions 



Relationships and Learning Activities 


Not True/Rarely True 
Little/No Evidence 


Sometimes True 
Some Evidence 


Usually/Always True 
Consistent Evidence 


Not Applicable/ 
Not Observed 


Caregiver With Children--Using Positive Guidance and Discipline 


11 


Caregiver states needed limits and 
consequences clearly and calmly. 


1 


2 


3 




12 


Caregiver helps children learn to understand/ 
and express feelings and gain control over 
impulses. 


1 


2 


3 




13 


Caregiver helps children to notice others’ 
needs and feelings. 


1 


2 


3 




14 


Caregiver helps children experience the 
consequences of their own misbehavior when 
safe. 


1 


2 


3 




15 


Caregiver does not force children into 
activities except during transitions. 


1 


2 


3 




16 


Caregiver emphasizes what children should do 

rather than what they should not do. 


1 


2 


3 




17 


Caregiver encourages/helps children resolve 
conflicts by talking through feelings and 
solutions. 


1 


2 


3 




18 


Caregiver redirects children or introduces a 
new activity to minimize their frustrations. 


1 


2 


3 




19 


Any time out or time away from the group is 

used to help child gain self-control, not as 
punishment. 


1 


2 


3 





1 Needed limits/consequences rarely/never stated OR if stated, not in calm, clear way 

2 Needed limits only sometimes stated clearly and calmly 

3 Needed limits consistently stated clearly and calmly 

Rarely/never helps children understand/express feelings or control impulses by 
talking 

2 I Sometimes helps children understand/express feelings OR only helps some children 

3 | Consistently helps children understand/express feelings and control impulses by 

talking through feelings, giving alternative strategies for expressing feelings or 
gaining control 

1 Rarely/never helps children notice others’ needs and feelings 

2 Sometimes helps children notice others’ needs, feelings OR only helps some children 

3 Consistently helps children notice others’ needs and feelings 

Rarely/never lets children experience consequences of misbehavior; imposes 
arbitrary limits 

2 I Sometimes helps children experience natural consequences OR only some children 

3 | Consistently helps children experience natural consequences of their own 

misbehavior when it is safe to do so. 

1 Often forces children into activities, doesn’t let them make choices on own 

2 Sometimes forces children into activities rather than letting them decide OR only 

3 some 

Consistently lets children decide their own activities Do not count transitions 

1 Instructions usually/consistently emphasize what children should not do 

2 Some instructions emphasize what to do and others what not to do 

3 Instructions usually focus on what children should do 

1 | Rarely/never helps children resolve conflicts by talking through feelings and solutions; 
resolves conflicts on her own without involving children 

2 | Sometimes helps children resolve conflicts by talking; other times solves them herself 
OR only helps some children 

3 | Consistently helps resolve conflicts by talking rather than resolving conflicts without 

input from children. 

Rarely/never helps minimize children’s frustration by redirecting/introducing new 
activity 

Sometimes helps children by redirecting/offering new activity OR only helps some 
Consistently minimizes children's frustration by redirecting/introducing new activity 



Time out used as punishment or time out lasts too long 

Timeout sometimes used as punishment and sometimes to help child regain control 
Timeout consistently used in positive manner and for appropriate length of time. 





Usually/Always True 
Consistent Evidence 



Definitions 



1 There are substantial amounts of time when caregiver cannot see or hear children 
(excluding children over 6 years) 

2 There are a few times when caregiver cannot see or hear all children (excluding 
children over 6 years) 

3 Children 6 years and under can be see or heard at all times etc; there is no time 
when caregiver is out of range of either seeing or hearing what children are doing 

• Children 2 years or under must always be in line of sight when outside 

• Children 6 years or under must be in sight or hearing at all times 

« Other arrangements may be made for children 6 years or older 

1 Supervision is often insufficient for each child’s activities and abilities 

2 Supervision is sometimes sufficient, but there are times/activities when there should 
be more supervision (e.g., depending on where children are, materials they are using) 

3 Supervision is consistently appropriate for each child’s activities and abilities 

1 No additional supervision during potentially hazardous activities 

2 Additional, appropriate supervision observed during some but not all potentially 
hazardous activities 

3 Additional, appropriate supervision consistently provided during potentially hazardous 

activities. 

1 One or more children are left in restraining equipment for more than 30 minutes in a 
2-hour period when they are awake 

2 One or more children are left in restraining equipment for more than 20 minutes in a 
2-hour period when they are awake 

3 Children who are awake are not left in restraining equipment for more than 20 
minutes in any 2-hour period 




0 

13 
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0 

o 



Caregiver With Children — Does No Harm 


24 


Caregiver does not use physical punishment or 

humiliation. 


1 


2 


3 




25 


Caregiver does not handle children roughly. 


1 


2 


3 




26 


Caregiver does not criticize, shame, tease, 
threaten, or yell at children. 


1 


2 


3 




27 


Caregiver does not take over or dominate play 
or exert unnecessary control. 


1 


2 


3 




28 


Caregiver avoids power struggles with children. 


1 


2 


3 






Usually/Always True 
Consistent Evidence 



Definitions 



1 Physical punishment or humiliation used two or more times or with two children 

2 Physical punishment or humiliation observed one time 

3 No use of physical punishment (hitting, slapping, smacking, spanking, pinching) or 

_ humiliation (criticize, speak degradingly or sarcastically). 

1 Multiple children handled roughly (yanking, pulling, etc) OR one child handled roughly 
two or more times 

2 One child handled roughly once 

3 No children handled roughly 

1 Multiple children/groups of children criticized, shamed, teased, threatened, or yelled 
at OR one child multiple times 

2 One child/group of children criticized, shamed, teased, threatened, or yelled at once 

3 No children are criticized, shamed, teased, threatened, or yelled at 

1 Tries to dominate or exert unnecessary control over children more than once 

2 Tries to dominate or exert unnecessary control over children once 

3 Does not try to dominate or exert unnecessary control over children 

(power struggle=ongoing competition for power where each person tries to 
control/subdue the other) 

1 Two or more power struggles or an ongoing struggle with at least one child 

2 Power struggle with one child. 

3 Power struggles with children consistently avoided. Tries to help children understand 

why they need to do something they don’t want to do. 





Usually/Always True 
Consistent Evidence 



Definitions 



1 Provides few (0-1 ) opportunities and little/no encouragement for children to work/play 
together (other than meal times) 

2 Provides some opportunities and encouragement for children to work/play together 
OR only encourages some children (other than meal times) 

3 Provides many opportunities and encouragement (other than meal times) for children 
to do things together, not necessarily structured group activities 

1 No/almost no discussion/teaching children about sharing, taking turns, cooperating 

2 A few discussions/instances of teaching children about sharing, taking turns, and 
cooperating OR only teaches some children 

3 Many discussions/instances of teaching children about sharing, taking turns, and 

_ cooperating, not just structured lessons but also using everyday situations 

1 Rarely/never teaches, encourages children to take care of others 

2 Sometimes teaches and encourages children to take care of others but misses some 
opportunities, especially if there are children of different ages, levels of competence 

3 Uses many opportunities to teach and encourage children to take care of others, 
especially older and more competent children 

1 Rarely/never teaches social rules or limits to children 

2 A few instances of teaching social rules or limits but misses some opportunities OR 
only teaches some children 

3 Takes many opportunities to teach social rules or limits tomost/allchildren, not only in 

structured discussions but also in the context of everyday activities (e.g., on field 
trips, greeting visitors) 

1 Ignores teasing, bullying; rarely/never children stand up for each other/themselves 

2 Sometimes helps children notice teasing etc and to stand up for each other, but 
ignores/fails to notice some of this behavior OR only helps some children 

3 Consistently helps children notice teasing etc and stand up for each other/ 

themselves and helps assure fair outcomes 

1 Rarely/never teaches children about community members, community roles 

2 Takes some opportunities (1-2) to teach children about community members, roles 
OR only teaches some children 

3 Takes many opportunities to teach children about community members, roles 

1 Rarely/never responds factually to/ignores children’s questions/comments about 
differences 

2 Sometimes responds factually to children’s questions/comments about differences 
but other times either ignores children’s curiosity or responds inappropriately OR only 
responds to some children 

3 Consistently responds factually and respectfully to children’s curiosity about people’s 

similarities and differences 

1 Children rarely/never expected to take care of materials, etc, including not destroying 
them and cleaning up 

2 Children only sometimes expected to take care of materials or only some children 
expected to take care 

3 Consistently teaches children to take care of the equipment etc; takes advantage of 
opportunities during activities 






Usually/Always True 
Consistent Evidence 



Definitions 



1 Rarely/never gives children opportunities to plan and freely choose own activities; 
always provides children with activities that they have to do 

2 Sometimes gives children opportunities to plan/choose own activities but sometimes 
requires them to follow adult-directed schedule 

3 Consistently/regularly gives children opportunities to plan and select own activities, 

providing support by helping children take turns in activities, try new activities, have 
time for activities they want to do 

1 Children rarely/never given sufficient free-play time (< 60 minutes) over a half-day 
AND insufficient materials/activities to choose among 

2 Children are given either sufficient free-play time (at least 60 minutes) during a half- 
day period) OR sufficient materials/activities to choose among, but not both 

3 Children are given sufficient free-play time (at least 60 minutes) during half-day AND 
sufficient materials/activities to explore their own interests 

1 Rarely/never supports children’s play; either ignores what children are doing or tries 
to exert control over children’s play by offering unneeded suggestions, changing 
directions 

2 Sometimes supports children’s play but at other times either ignores children, tries to 
control children’s play 

3 Consistently supports children’s play by observing and providing encouragement and 

praise or by joining in; does not try to take over the play activity with suggestions 
unless asked 

1 No opportunities for active physical play provided, either outdoors or indoors 

2 Opportunities for active physical play, but lasting less than 45 minutes OR only for 
some children 

3 Opportunities for physical play that last at least 45 minutes 





Usually/Always True 
Consistent Evidence 



Definitions 



1 Rarely/never takes advantage of or builds on natural learning experiences; misses 
opportunities 

2 Sometimes takes advantage of/builds on natural learning experiences, but also 
misses some opportunities OR only builds on natural experiences for some children 

3 Consistently takes advantage of and builds upon natural learning experiences and 

“teachable moments" 

1 Frequent instances where children do not have an activity in which to engage; does 
not offer alternatives for children 

2 Some instances when one or more children do not have an activity in which to 
engage and alternatives only sometimes provided 

3 Children consistently have an activity in which to engage and alternative activities 

provided when needed 

1 Rarely/never uses open-ended questions with children 3 years and older 

2 Only a few (1 -2) open-ended questions or only to some of children 3 yrs and older 

3 Multiple instances of open-ended questions about what they are doing and thinking; 
questions asked of most/all children 3 years and older 



1 Rarely/never helps children learn specific skills and concepts when they are ready 

2 Sometimes helps children learn specific skills but does not always respond when 
children indicate they are ready or only some children are helped 

3 Consistently helps children learn specific skills and concepts when they are ready, 

including adaptive and cognitive skills 

1 Children are not introduced or oriented to what is involved in new activities. 

2 Children only sometimes are oriented to new activities or only some children are 
given this support 

3 Children are consistently introduced or oriented to what is involved in new activities 





Usually/Always True 
Consistent Evidence 



Definitions 



No/almost no opportunities/resources provided for fine motor play for any age 
Limited number of opportunities/resources provided for fine motor play OR 
opportunities only for some children 

Multiple opportunities/resources provided for fine motor play for children of all ages 

No opportunities/resources provided for dramatic play for children (costumes/clothing, 
household items, telephones, etc.) 

Limited opportunities/resources provided for dramatic play OR opportunities only for 
some children 

Multiple opportunities/resources provided for dramatic play for children of all ages 
No teaching of early math concepts to children of any age — comparisons, shapes, 
measurement, etc. 

Limited teaching of early math concepts OR only to some children 
Multiple instances of teaching of early math concepts to children of all ages 



Little/no encouragement of children to use math in everyday contexts. 

Some encouragement of children to use math in everyday contexts but misses some 

opportunities OR only with some but not all children 

Consistently encourages most/all children to use math in everyday contexts 



No/almost no opportunities/resources provided for construction activities for children 
of any age 

Limited number of opportunities/resources provided for play OR opportunities only for 
some children 

Multiple opportunities/resources provided for fine motor play for children of all ages 
No/almost no opportunities provided for children of any age to explore natural world 
Limited number of opportunities provided for exploring natural world OR opportunities 
only for some children 

Multiple opportunities provided for children of all ages to explore natural world 

No/almost no creative activities provided or all creative activities are close-ended and 
caregiver-directed. 

Some creative activities are open-ended and self-directed, others are not. 

All/most creative activities are open-ended and child-directed 

No examples of children's art visible/available 

Only a few examples of children's art visible/available OR there is a clear preference 
for realistic or “pretty” work 

Multiple examples of children's art visible/available and all children's art treated 
equally (no preference for pretty or realistic work) 





Usually/Always True 
Consistent Evidence 



Not Applicable/ 
Not Observed 



Definitions 



1 No opportunities/resources for children of any age to make their own music 

2 Single opportunity/resource for children to make their own music OR only some 
children provided with an opportunity to make music 

3 Multiple opportunities/resources for children of all ages to make music, chants, using 
voices/instruments 

1 No opportunities for children of any age to dance or move creatively 

2 Single opportunity for children to dance or move creatively OR only some children 
provided with an opportunity to dance 

3 Multiple opportunities for children of all ages to dance or move creatively 

1 No opportunities/resources for children of any age to engage in sensory play (water, 
clay, sand) 

2 Single opportunity/resource for children to engage in sensory play OR only some 
children provided with an opportunity/resources for sensory play 

3 Multiple opportunities/resources for children of all ages to engage in sensory play 




Relationships and Learning Activities 


Not True/Rarely True 
Little/No Evidence 


Sometimes True 
Some Evidence 


Usually/Always True 
Consistent Evidence 


Not Applicable/ 
Not Observed 




Definitions 


Supporting Language Development and Early Literacy 




57 


Caregiver reads to all children every day. 


1 


2 


3 




1 

2 

3 


No reading aloud to any children 

Reads no more than one book (or reads for less than 30 minutes) daily to children of 
any age OR reads multiple books but only to some children 
Reads at least 30 minutes daily with most/all children of all ages (alternative 
arrangements can be made for school-age children)* 


58 


Caregiver encourages children to look at or 
read books or other forms of print on their own. 


1 


2 


3 




1 

2 

3 


Rarely encourages children to look at or read books on their own. Books are often 
not available; reading is rarely suggested 

Sometimes encourages children to look at or read books on their own. Books are 
only available some of the time 

Consistently encourages children to look at or read books on their own. Books are 
available to children a large portion of the time; reading on own is often suggested 


59 


Caregiver draws children’s attention to the 

functions and features of print. 


1 


2 


3 


No children 
over 2 years 


1 

2 

3 


Never calls attention to functions/features of print 

Sometimes calls attention to functions/features of print through high-quality activities 
OR only for some children 

Consistently calls attention to functions of print (labeling, naming categorizing, 
describing) or features of print (directionality, capital versus lower case letters, spaces 
between words) through high-quality activities 


60 


Caregiver draws children’s attention to the 
sounds in words, talks about how letters and 
letter combinations sound. 


1 


2 


3 




1 

2 

3 


Never calls attention to sounds of words over the day for any children 
Sometimes calls attention to sounds of words over the day using high quality 
techniques OR only with some children 

Consistently provides opportunities for most/all children to learn about sounds of 
letters/words using high-quality techniques- text with rhymes/alliteration, songs, 
games that emphasize rhyming syllables in words (e.g., clapping out syllables) 


61 


When reading to children, caregiver engages 
them in discussion, asks open-ended 
questions. 


1 


2 


3 




1 

2 

3 


Reading aloud: no open-ended questions or extended discussion about book 
Reading aloud: some questions but usually closed-ended or only require short/one- 
word answers 

Reading aloud: consistently engages most/all children in discussion, asks open- 
ended questions* 


62 


Caregiver engages children in language games, 
rhymes, songs. 


1 


2 


3 




1 

2 

3 


Never engages children in language games, rhymes, songs 

Sometimes engages children in language games, rhymes, songs or only does this 

with some children 

Engages most/all children in multiple language games, rhymes, songs* 


63 


Caregiver encourages children to write letters or 
words. 


1 


2 


3 


No children 
over 2 years 


1 

2 

3 


Does not encourage children to write letters/words. Any writing is done by caregiver. 
Sometimes encourages children to write letters/words, but activities are lower quality 
(tracing, copying, stencils — insisting on conventional letter formation or spelling) 
Consistently encourages most/all children to write letters/words, including emergent 
writing, captioning, dictation with caregiver, writing own name on work, bookmaking, 
journals * 



'Note: In setting with 4 or fewer children, code 3 only if caregiver does this with all children; 
in setting with 5 children, code 3 if caregiver does this with at least 4 of 5 children; 
in setting with 6 children, code 3 if caregiver does this with at least 4 of 6 children; 
in setting with 7 or 8 children, code 3 if caregiver does this with at least 5 children. 





Usually/Always True 
Consistent Evidence 



Definitions 



1 Rarely/never encourages use of language to communicate ideas/ thoughts in different 
activities. Most adult/child verbal interactions short, about management issues 

2 Limited encouragement of use of language to communicate ideas/ thoughts in 
different activities. Many adult/child verbal interactions short, about management . 

3 Consistently encourages use of language to communicate ideas and thoughts over 

and in different types of high-quality activities-extended discussions; higher-level 
cognitive operations such as prediction, explanation, developing hypotheses. 

1 ELLs segregated from non-ELLs; ELLs discouraged from joining conversations with 
non-ELLs. Does not appear positive about having ELL children in home. 

2 ELLs sometimes segregated from non-ELLs; ELLs sometimes discouraged from 
joining conversations with non-ELLs. Only sometimes appears positive about having 
ELL children in home 

3 Intentionally includes ELL children in activities; ELLs encouraged to join 

conversations with non-ELLs. Consistently positive about having ELL children 

1 Discourages ELLs from using native language with each other or shows disapproval 
when ELLS use their native language 

2 Allows but does not actively encourage children to use their native language with 
each other. 

3 Consistently encourages ELL children to talk to each other in their native language 

1 Does not use both English/ELL language in literacy activities. No bilingual materials 

2 Sometimes uses ELL language and English in literacy activities. Some materials 
used in literacy activities are bilingual and age-appropriate 

3 Consistently integrates ELL language/English in literacy activities. Most/all materials 
used in literacy activities are bilingual and are attractive, age-appropriate 





Usually/Always True 
Consistent Evidence 



Not Applicable/ 
Not Observed 



Definitions 



1 One or more children watch TV or videos for more than V 2 hour, or use computers for 
more than 20 minutes AND no alternative activities available for non-watchers/users. 

2 Either one or more children watch TVs or videos more than V 2 hour (or 1 full length 
movie) or use computer more than 20 minutes OR watching is less than 1 hour but no 
alternative activities available for non-watchers/users. 

3 TV or video watching is less than 1/2 hour (or one full-length movie) or computer use 

is less than 20 minutes for all children and alternate activities are available for non- 
watchers/users. 

1 One or more TV programs, movies, computer/video games are violent, stereotyped, 
sexually explicit 

2 One TV program, movie, computer/video game is not appropriate for the age of 
children watching it, but not violent 

3 No TV program, movie, computer/video game is inappropriate for the age of children 

watching it-violent, stereotyped, sexually explicit OR no TV, videos, computer, video 
games observed. 




Summary Rating of Caregiver 






Arnett Global Rating Scale of 
Caregiver Behavior 



03 



Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 



1 Speaks warmly to children (e.g., positive 
tone of voice, body language). 

2 Seems critical of children (e.g., puts 
children down, uses sarcasm). 

Listens attentively when children speak 

3 to her (e.g., looks at children, nods, 
rephrases their comments, engages in 
conversations). 

Places high value on obedience (e.g., 

4 expects children to follow adult agenda, fails 
to respond to daily events in a flexible way). 



2 3 



4 

[critical 

4 

[listens] 



Seems distant or detached from children 

(e.g., sits apart, does not touch children, 
does not greet children). 



2 3 4 

[distant] 



Seems to enjoy children (e.g., conveys 
5 warmth by smiling, touching, taking 
children's conversations seriously). 

Reasons with children when they 

7 misbehave (e.g., explains rule they are 
breaking, discusses consequences, 
redirects behavior, discusses alternatives). 

Encourages children to try new 

8 experiences (e.g., suggests children do it 
together, helps children start, introduces 
new materials). 

Exercises too much control over children 

9 (e.g., doesn’t seek child input, rigidly 
adheres to rules and schedules). 



2 3 



4 

[enjoys] 



4 

[control] 



Speaks with irritation or hostility to 
children (e.g., sharp tone, raises voice). 



CM CO ^ 



Definitions 



1 Rarely speaks warmly to children; neutral or negative 

2 Sometimes speaks warmly to children or only speaks warmly to some children 

3 Speaks warmly to children much of the time or to some children 

4 Consistently speaks warmly to children 

1 Never seems critical of children, verbally or non-verbally 

2 Rarely seems critical of children, verbally or non-verbally 

3 Often seems critical of children, verbally or non-verbally 

4 Consistently Is critical of or sarcastic with children, verbally or non-verbally 

1 Does not listen attentively when children speak 

2 Rarely listens attentively when children speak 

3 Usually listens attentively when children speak 

4 Consistently listens attentively when children speak — looks at children, talks with them 

1 Does not seem to place a high value on obedience Responds to daily events flexibly 

2 Sometimes expects children to follow rules/ conventions but allows some flexibility 

3 Usually emphasizes following rules or conventions rather than responding flexibly 

4 Consistently expects children to follow adult rules/conventions Does not respond to daily 

events flexibly 

Consistently is involved with, watches children with interest 

Is usually involved with children, but in some instances is distant or detached 
Is usually distant or detached but not always 

Is consistently distant or detached Sits apart, does not touch /talk to children 

1 Consistently fails to show enjoyment of/warmth toward children-may be harsh or distant 

2 Usually fails to show enjoyment of/warmth toward children 

3 Usually seems to enjoy children but sometimes is distant or harsh 

4 Consistently seems to enjoy children — warm, smiling, touching, listening seriously 

1 Rarely explains rules/discusses consequences when children break them 

2 Usually does not explain rules, discuss consequences 

3 Usually explains the reasons for rules and consequences for breaking them 

4 Consistently explains the reasons for rules, explains consequences, redirects behavior 

1 Does not encourage children to try new experiences 

2 Rarely encourages children to try new experiences 

3 Often encourages children to try new experiences 

4 Consistently encourages children to try new experiences-provides opportunities/support 

1 Consistently seeks child input, does not rigidly adhere to rules and schedules 

2 Imposes some control over children’s behavior but usually allows children flexibility 

3 Keeps children in control most of the time-imposes rules and manages children’s behavior 
closely 

4 Exercises too much control over children Adheres rigidly to rules/schedules with no child input 

1 Does not speak with irritation or hostility to children; does not uses sharp tone or raise voice 

2 Rarely speaks with irritation or hostility to children 

3 Often speaks with irritation or hostility to children 

4 Consistently speaks with irritation and/or hostility to children; uses sharp tone, raises voice 




Arnett Global Rating Scale of 
Caregiver Behavior 


Not at all 


Somewhat 


Quite a bit 


Very much 




Definitions 


Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 




11 


Seems enthusiastic about children’s 
activities and efforts (e.g., congratulates 
children, states appreciation for efforts). 


1 


2 


3 


4 

[enthuse] 


1 

2 

3 

4 


Does not seems enthusiastic about children’s activities/efforts 
Rarely seems enthusiastic about children's activities/efforts 
Usually seems enthusiastic about children’s activities/efforts 

Consistently seems enthusiastic about children’s activities/efforts; congratulates them, 
recognizes their efforts 


12 


Threatens children in trying to control 
them (e.g., uses bribes and threats of 
punishment). 


1 


2 


3 


4 

[threats] 


1 

2 

3 

4 


Does not threaten punishment/bribe children to control/manage them 
Usually does not bribe/threaten punishment to control children 
Often uses bribes/threats of punishment to control children 

Consistently uses bribes/threats of punishment to control children 


13 


Spends considerable time in activity not 
involving interaction with children (e.g., 
does adult tasks during child activity 
periods). 


1 


2 


3 


4 

[apart] 


1 

2 

3 

4 


Rare spends time in activities that do not involve children 
Usually remains involved with children during 
Frequently spends time in activities that do not include children 
Spends substantial time in activities that do not include children 


14 


Pays positive attention to children as 
individuals (speaks to individual children, 
uses names, calls attention to prosocial 
behaviors, comments on their strengths). 


1 


2 


3 


4 

[attends] 


1 

2 

3 

4 


Does not pav positive attention to children as individuals; does not sinale out children 
Rarelv oavs positive attention to children as individuals 

Often oavs positive attention to children as individuals or oavs attention to some but not all 
children 

Consistently pays positive attention to most/all children as individuals 


15 


Reprimands children too strongly when 
they misbehave (e.g., fails to acknowledge 
difficulties of learning self-control, fails to 
redirect behavior). 


1 


2 


3 


4 

[repri- 

mands] 


1 

2 

3 

4 


Rarely/never reprimands children too strongly Always tries to redirect children’s behavior 
without punishing them 

Sometimes reprimands children too strongly but sometimes deals with misbehavior by talking 
to children, offering alternatives, asking about reasons 
Often reprimands children who misbehave too strongly 

Consistently reprimands children too strongly-uses punishment, reprimands, threats 


16 


Talks to children on a level they can 
understand (e.g., uses terms familiar to 
children, checks for clarification). 


1 


2 


3 


4 

[talks] 


1 

2 

3 

4 


Consistently talks to children in a way that is inappropriate for their developmental level 
Usually talks to children on a level they cannot understand 

Usually talks to children on a level they can understand, altering language for different levels 
Consistently talks to children at a level they can understand, checks to be sure they 
understand 


17 


Punishes children without explanation 

(e.g., does not discuss infraction). 


1 


2 


3 


4 

[punish] 


1 

2 

3 

4 


Does not punish children without explanation-discusses misbehavior, reason for rules 
Usually provides explanation for punishment and discusses misbehavior, rule 
Usually fails to provide explanations to children for punishment 
Consistently punishes children without explanation for punishment, discussion 


18 


Exercises firmness when necessary (e.g., 
provides clear and direct directions, checks 
for understanding). 


1 


2 


3 


4 

[firm] 


1 

2 

3 

4 


Consistently lets misbehavior, conflicts get out of control without exercising firmness-does not 
provide clear direction 

Often lets misbehavior, conflicts get out of control without stepping in 

Usually does not let misbehavior/conflicts get out of control without stepping in, exercising 

firmness 

Consistently exercises firmness, direction when necessary — doesn’t let things get out of 
control 


19 


Encourages children to exhibit prosocial 
behavior (e.g., sharing, cooperating, pairs 
socially skillful with those children who need 
practice). 


1 


2 


3 


4 

[share] 




1 

2 

3 

4 


Does not encourage prosocial behavior (sharing, cooperation) 

Rarely encourages prosocial behavior 
Usually encourages prosocial behavior 

Consistently takes advantage of opportunities to encourage prosocial behavior; encourages 
children to work together, help each other 




Arnett Global Rating Scale of 
Caregiver Behavior 






Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 

20 Finds fault easily with children (e.g., uses 1 2 3 

negative tone, is critical). P 

Fails to show interest in children’s 

21 activities (e.g., removes self from children’s 1 2 3 

activities, doesn’t talk to children or extend 

their conversations). in1 

Seems to prohibit many of the things 

22 children want to do (e.g., adheres to rigid 1 2 3 

schedule or adult outcomes and agendas). [P r 

23 Fails to supervise children very closely 1 2 3 

(e.g., fails to foresee and forestall mishaps). 



Expects children to exercise self-control 

(expects children to be undisruptive for short 
group, caregiver-let activities; to be able to 
stand in line calmly; reminds children of 
expectations, asks for cooperation in 
supportive ways). 

When talking to children, kneels, bends 
or sits at their level to establish better 
eye contact (e.g., e.g., ensures connection 
when having a conversation). 

Seems unnecessarily harsh when 
scolding or prohibiting children (e.g., 
angry tone, shakes children, uses physical 
punishment, uses “time out” without 
explanation). 

Does self-help tasks for children (e.g., 
dresses them, wipes their noses, selects 
materials for choice time). 



^ co m -*■ -f^corv) -*■ -f^coro 



Definitions 



1 Consistently avoids finding fault easily with children; does not blame or criticize needlessly 

2 Rarely finds fault easily with children, blames or criticizes needlessly 

3 Often finds fault with children too easily, blames or criticizes needlessly 

4 Consistently finds fault too easily with children Is quick to criticize, lay blame, 

1 Consistently shows interest in children’s activities, talking to/interacting with them 

2 Often shows interest in children’s activities but sometimes does not or not for some children 

3 Often fails to show interest in children's activities or interacts with them in activities, 

4 Consistently fails to show interest in children's activities or to interact with them in activities 

1 Does not prohibit most things children want to do — consistently lets them decide what to do 

2 Rarely prohibits what children want to do; usually lets them decide 

3 Often prohibits what children want to do 

4 Consistently prohibits what children want to do Follows adult agenda rigidly 

1 Consistently supervises closely, rarely letting children out of sight or hearing; foresees mishaps 

2 Usually supervises children closely; usually forestalls and foresees mishaps 

3 Often fails to supervise children very closely Often fails to foresee mishaps 

4 Consistently fails to supervise very closely, often leaving children out of sight or hearing; fails 

to foresee mishaps 

1 Does not expect children to control their own behavior-to be able to stand in line, wait their 
turn, listen and participate in an orderly way during caregiver-led activities 

2 Usually does not expect children to be in control of their own behavior 

3 Usually expects children to be in control of their own behavior 

4 Expects children to exercise self-control; asks for cooperation in supportive ways 



1 | Does not kneel, bend when talking to children; does not ensure a connection 
Rarely kneels, bends when talking to children 

Usually kneels, bends when talking to children to ensure a connection 

Consistently moves to child’s height when talking to ensure a connection; establishes eye 

contact 

Consistently avoids unnecessary harshness in discipline; does not use angry tone, use 
physical punishment, use “time out” without explanation 
Usually avoids unnecessary harshness when disciplining children 
Often seems unnecessarily harsh when scolding children 

Consistently seems unnecessarily harsh when scolding or prohibiting; uses angry tone, 

physical punishment, “time out” without explanation 

Consistently lets children do own self-help tasks-wipe own noises, wash own hands, dress for 
outdoors, select materials for choice time. 

Often lets children do own self-help tasks. 

Often does self-help tasks for children. 

Consistently does self-help tasks for children 




Arnett Global Rating Scale of 
Caregiver Behavior 


Not at all 


Somewhat 


Quite a bit 


Very much 


Circle one number for each item that best represents the extent are 
each of the following statements characteristic of this caregiver: 


28 


Does not get child assistance with tasks 

(e.g., watering plants, animal care, putting 
away toys, cleaning tables and other routine 
maintenance). 


1 


2 


3 


4 

[no 

assist] 


29 


Promotes leadership skills (e.g, gives 
assignments for jobs like line leaders, clean- 
up inspector, talks about children’s 
contributions to the group). 


1 


2 


3 


4 

[leaders] 


30 


Assists children in making productive 
choices (e.g., uses a planning process or 
discussion to outline choices during activity 
periods, narrows choices when children 
have difficulty). 


1 


2 


3 


4 

[choices] 



Definitions 



1 Consistently gets children to help with tasks 

2 Usually gets children to help with tasks 

3 Rarely gets children to help with tasks 

4 Does not get children to help with tasks--water plans, care for pets, put away toys/materials 



1 

2 

3 

4 



1 

2 

3 

4 



Does not promote leadership skills in children-does riot assign jobs, talk about children’s 
contributions to group. 

Rarely promotes leadership skills in children 
Often promotes leadership skills in children. 

Consistently promotes children's leadership skills through job assignments, talk about 
contributions to group, asking children’s opinions 

Does not assist children in making productive choices by discussing choices, helping narrow 
choices, redirecting poor choices 

Usually does not assist children in making productive choice. 

Usually assists children in making productive choices. 

Consistently assists children in making productive choices. 




